1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
|
---
title: Notes on setting up a FreeBSD home server
---
# Notes on setting up a FreeBSD home server
## 2018-09-17: Prologue
A few months ago, I purchased a beefy second-hand tower to act as a home server.
I was looking to bring some of the services that I was previously outsourcing
into a single location, and to expand my familiarity with networking and systems
administration. Specifically, I wanted to:
- Replace the small DigitalOcean box that I was using as a VPN/proxy when I
needed to use public WiFi
- Stop paying for a GitHub subscription to host private repositories
- Have a better home media and file sharing/backup solution
- Host a Minecraft server (nothing too serious, I occasionally play with a few
friends)
- Have a stable home for various VMs that I spin up as part of my security lab
(I've been playing around with pen testing and trying to learn more about
Windows as a part of this).
My initial solution was to install a free version of VMWare ESXi as a hypervisor
and create several virtual machines. It was actually quite easy to get ESXi up
and running and start creating VMs. For the past several months, my home network
has been completely routed through the server (it has dual Ethernet, so I'm
using pfSense in a VM as my firewall/NAT/DHCP/etc), and I've spun up several VMs
(mostly Ubuntu) for things like Gitlab and Minecraft.
However, there are a few things that I don't quite like. I did have an incident
following a power outage after my free trial of ESXi had expired but before I
inputted my free license key in the UI. This resulted in my pfSense VM not
auto-booting and due to some poor configuration on my part, I was unable to
access the ESXi web UI to enter the license key without resetting the network
settings through the ESXi console. This brings me to my second gripe: the ESXi
web UI is _very_ buggy and overall pretty awful to use. Certain pages have to be
reloaded to work properly, dialogs are randomly empty, etc. Thirdly, I've found
myself creating a "general purpose" VM that I can SSH into remotely. While
there's nothing explicitly _wrong_ with this, it just doesn't feel quite right
to me to have a general purpose server that is completely parallel to my other
server VMs.
As a result of these shortcomings and learnings, I have decided to embark upon a
journey towards further simplification and reliability. I'll be replacing ESXi
with FreeBSD, a rock-solid operating system. Rather than running a utility VM,
I'll simply have the FreeBSD system on the server itself as a "base of
operations."
I plan to learn more about and use several tools during this process. Currently,
I only have one 2 TB drive installed. I plan to add a second one and use zfs to
create a mirrored vdev pool for redundancy. This will make me feel a lot better
about using my server as a backup destination. Of course, this in itself is not
a complete backup solution, but it's a significant step forward from just
relying on a single disk. Rather than running pfSense in a VM, I plan to just
use the ISC DHCP server from the ports collection and use the built-in `pf`
firewall to accomplish just about everything I was using pfSense for. I'll
likely also end up running a BIND DNS server for a few local network things.
I am still learning about jails in FreeBSD, but I think they could replace a few
of the VMs I have currently, such as the Minecraft and GitLab servers. I plan to
use bhyve to run things like Windows VMs for pen testing that jails are clearly
not suited for.
I've used FreeBSD as my desktop OS in the past, and really love how it feels
compared with GNU/Linux. Everything just seems more straightforward, and I was
surprised to find that things like graphics drivers Just Work™ under
FreeBSD where they require a lot of ugly finagling under Linux. I'm quite
looking forward to using FreeBSD more often frequently, and gaining more depth
in some of its great tools like jails and pf.
To start making the transition (which might be a little painful), I've installed
a fresh copy of FreeBSD 11.2 on a currently-unused machine to start poking
around with zfs configurations, jails, and bhyve. This will give me the
foundation I need to effectively set up my top-level environment and hopefully
get it mostly right the first time. Incidentally, I'm also about half way
through reading [The Book of PF](https://nostarch.com/pf3) from No Starch Press,
which will no doubt be helpful in my transition from pfSense to pure pf.
I intend to update this page with notes as I continue on my FreeBSD journey.
Stay tuned!
## 2018-09-20: Experiment 1: Jails
In my preparations for removing ESXi, I tried creating a simple jail on my test
box `helios`. As part of my purpose is to learn as much as possible, I decided
against using a tool like `ezjail` in favor of doing it "by hand." While the
FreeBSD Handbook has some information on creating jails without using additional
tools, pretty much every other document I found suggested using ezjail. There's
a chance I'll revisit ezjail in the future, as it seems to have some helpful
features like having a "base jail" so you only need one copy of the FreeBSD base
system, but for now I'd like to do as much as possible without additional tools.
My goal for this experiment was to set up a simple web server (nginx) inside a
jail. To start, I edited `/etc/jail.conf` to contain the following:
```
www {
host.hostname = www.local;
ip4.addr = 10.0.2.202;
path = "/usr/jail/www";
exec.start = "/bin/sh /etc/rc";
exec.stop = "/bin/sh /etc/rc.shutdown";
}
```
Next, I used `bsdinstall(8)` to install the base system instead of compiling
from source:
```
root@helios:~ # bsdinstall jail /usr/jail/www
```
I then added `jail_enable="YES"` to `/etc/rc.conf` and started the jail:
```
root@helios:~ # service jail start www
```
This took a few seconds to complete, and then the jail showed up when I ran
`jls`:
```
root@helios:~ # jls
JID IP Address Hostname Path
1 10.0.2.202 www.local /usr/jail/www
```
I was able to enter the jail:
```
root@helios:~ # jexec www /bin/sh
#
```
But I seem not to have Internet connectivity, as attempting to use `pkg-ng`
fails:
```
# pkg install nginx
The package management tool is not yet installed on your system.
Do you want to fetch and install it now? [y/N]: y
Bootstrapping pkg from pkg+http://pkg.FreeBSD.org/FreeBSD:11:amd64/quarterly, please wait...
pkg: Error fetching http://pkg.FreeBSD.org/FreeBSD:11:amd64/quarterly/Latest/pkg.txz: Non-recoverable resolver failure
A pre-built version of pkg could not be found for your system.
Consider changing PACKAGESITE or installing it from ports: 'ports-mgmt/pkg'.
```
Running `ifconfig` inside the jail shows that I do not seem to have an IP
address, nor can I seem to communicate with any hosts. Interestingly when I
attempt to ping my gateway, I get the message:
```
ping: ssend socket: Operation not permitted
```
Clearly there's something I've not yet figured out.
## 2018-10-13: Experiment 2: Jail Networking Continued
I decided to take another crack at the jail configuration I started in
[Experiment 1](#2018-09-20-experiment-1-jails). After reading bits and pieces of
a few random websites (including various ServerFault posts), on an inkling I
added the line `interface = "bge0";` to my `/etc/jail.conf` file and ran
`service jail restart www` (`bge0` is my LAN interface on the host). After
`jexec`ing in, I tried `pkg install nginx` again and it worked like a charm!
I also noticed that when I run `ifconfig` on my host now, both the original
10.0.2.201 and the jail's 10.0.2.202 addresses had been added to the `bge0`
interface. I wondered whether that meant that I could now SSH into the host
using the jail's IP address. So on my laptop, I ran `ssh bb@10.0.2.202` and lo
and behold, it worked. The opposite, however, is _not_ true: loading
http://10.0.2.201 in a web browser does not give me the beautiful "welcome to
nginx" page that http://10.0.2.202 has.
I'm sure some trickier stuff will arise when dealing with NAT and multiple
interfaces, but for now I'm satisfied that I have a basic understanding of how
to set up a service in a jail and expose it to the network.
|
