What have I neglected to do in order to get networking in a jail?

Discussion:

(too old to reply)

James B. Byrne via freebsd-questions

2018-05-30 15:24:03 UTC

On FreeBSD-11.1 host:

[***@host:~]# service pf onestatus
pf.ko is not loaded

In /etc/rc.conf
. . .
defaultrouter="216.185.71.1" # Gateway
gateway_enable="YES" # Enable as ipv4 LAN gateway for
guests/jails
#ipv6_gateway_enable="YES" # Enable as ipv6 LAN gateway

# Aliases on the host i/f are set here - jailed aliases are handled by
ezjail
ifconfig_vtnet0_alias0="inet 192.168.216.18 netmask 255.255.255.255"
#ifconfig_vtnet0_alias1="inet 192.168.216.xxx netmask 0xFFFFFFFF"
#ifconfig_vtnet0_alias2="inet 192.168.216.xxy netmask 0xFFFFFFFF"

### Enable and configure ezjail jails
# Setup the loopback interfaces that each jail will use
# Remember to add a 'set skip on lo#' clause in /etc/pf.conf
cloned_interfaces="lo1 lo2"
ipv4_addrs_lo1="127.0.31.1/32"
ipv4_addrs_lo2="127.0.32.1/32"

### Jailed Services
ezjail_enable="YES" # Enable ezjail jail manager

[***@host:~]# ifconfig

vtnet0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST>
metric 0 mtu 1500
options=80028<VLAN_MTU,JUMBO_MTU,LINKSTATE>
ether 58:9c:fc:0e:cd:bb
hwaddr 58:9c:fc:0e:cd:bb
inet 216.185.71.18 netmask 0xffffff00 broadcast 216.185.71.255
inet 192.168.216.18 netmask 0xffffffff broadcast 192.168.216.18
inet 218.185.71.31 netmask 0xffffffff broadcast 218.185.71.31
inet 192.168.216.31 netmask 0xffffffff broadcast 192.168.216.31
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
media: Ethernet 10Gbase-T <full-duplex>
status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
inet 127.0.0.1 netmask 0xff000000
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
groups: lo
lo1: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
inet 127.0.31.1 netmask 0xffffffff
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
groups: lo
lo2: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
inet 127.0.32.1 netmask 0xffffffff
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
groups: lo

[***@host:~]# jls
JID IP Address Hostname Path
1 127.0.31.1 mx31 /usr/jails/mx31

On jail:

***@mx31:~ # sysctl security.jail.allow_raw_sockets
security.jail.allow_raw_sockets: 1

***@mx31:~ # ifconfig
vtnet0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST>
metric 0 mtu 1500
options=80028<VLAN_MTU,JUMBO_MTU,LINKSTATE>
ether 58:9c:fc:0e:cd:bb
hwaddr 58:9c:fc:0e:cd:bb
inet 218.185.71.31 netmask 0xffffffff broadcast 218.185.71.31
inet 192.168.216.31 netmask 0xffffffff broadcast 192.168.216.31
media: Ethernet 10Gbase-T <full-duplex>
status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
groups: lo
lo1: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
inet 127.0.31.1 netmask 0xffffffff
groups: lo
lo2: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
groups: lo

***@mx31:~ # cat /etc/resolv.conf
search harte-lyne.ca
nameserver 216.185.71.33
nameserver 216.185.71.34
nameserver 127.0.0.1
options edns0

***@mx31:~ # cat /etc/hosts
# $FreeBSD: releng/11.1/etc/hosts 109997 2003-01-28 21:29:23Z dbaker $
#
# Host Database
. . .
#
#
::1 localhost localhost.harte-lyne.ca
127.0.0.1 localhost localhost.harte-lyne.ca

***@mx31:~ # pkg install bash
The package management tool is not yet installed on your system.
Do you want to fetch and install it now? [y/N]: y
Bootstrapping pkg from
pkg+http://pkg.FreeBSD.org/FreeBSD:11:amd64/quarterly, please wait...
pkg: Error fetching
http://pkg.FreeBSD.org/FreeBSD:11:amd64/quarterly/Latest/pkg.txz: No
address record
A pre-built version of pkg could not be found for your system.
Consider changing PACKAGESITE or installing it from ports:
'ports-mgmt/pkg'.

***@mx31:~ # ping 216.185.71.1
PING 216.185.71.1 (216.185.71.1): 56 data bytes
^C
--- 216.185.71.1 ping statistics ---
5 packets transmitted, 0 packets received, 100.0% packet loss

Why does this jail not have a network connection?
--
*** e-Mail is NOT a SECURE channel ***
Do NOT transmit sensitive data via e-Mail
Do NOT open attachments nor follow links sent by e-Mail

James B. Byrne mailto:***@Harte-Lyne.ca
Harte & Lyne Limited http://www.harte-lyne.ca
9 Brockley Drive vox: +1 905 561 1241
Hamilton, Ontario fax: +1 905 561 0757
Canada L8E 3C3

Herbert J. Skuhra

2018-05-31 08:55:19 UTC

Permalink

Post by James B. Byrne via freebsd-questions
pf.ko is not loaded
In /etc/rc.conf
. . .
defaultrouter="216.185.71.1" # Gateway
gateway_enable="YES" # Enable as ipv4 LAN gateway for
guests/jails
#ipv6_gateway_enable="YES" # Enable as ipv6 LAN gateway
# Aliases on the host i/f are set here - jailed aliases are handled by
ezjail
ifconfig_vtnet0_alias0="inet 192.168.216.18 netmask 255.255.255.255"
#ifconfig_vtnet0_alias1="inet 192.168.216.xxx netmask 0xFFFFFFFF"
#ifconfig_vtnet0_alias2="inet 192.168.216.xxy netmask 0xFFFFFFFF"
### Enable and configure ezjail jails
# Setup the loopback interfaces that each jail will use
# Remember to add a 'set skip on lo#' clause in /etc/pf.conf
cloned_interfaces="lo1 lo2"
ipv4_addrs_lo1="127.0.31.1/32"
ipv4_addrs_lo2="127.0.32.1/32"
### Jailed Services
ezjail_enable="YES" # Enable ezjail jail manager
vtnet0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST>
metric 0 mtu 1500
options=80028<VLAN_MTU,JUMBO_MTU,LINKSTATE>
ether 58:9c:fc:0e:cd:bb
hwaddr 58:9c:fc:0e:cd:bb
inet 216.185.71.18 netmask 0xffffff00 broadcast 216.185.71.255
inet 192.168.216.18 netmask 0xffffffff broadcast 192.168.216.18
inet 218.185.71.31 netmask 0xffffffff broadcast 218.185.71.31
inet 192.168.216.31 netmask 0xffffffff broadcast 192.168.216.31
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
media: Ethernet 10Gbase-T <full-duplex>
status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
inet 127.0.0.1 netmask 0xff000000
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
groups: lo
lo1: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
inet 127.0.31.1 netmask 0xffffffff
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
groups: lo
lo2: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
inet 127.0.32.1 netmask 0xffffffff
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
groups: lo
JID IP Address Hostname Path
1 127.0.31.1 mx31 /usr/jails/mx31
security.jail.allow_raw_sockets: 1
vtnet0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST>
metric 0 mtu 1500
options=80028<VLAN_MTU,JUMBO_MTU,LINKSTATE>
ether 58:9c:fc:0e:cd:bb
hwaddr 58:9c:fc:0e:cd:bb
inet 218.185.71.31 netmask 0xffffffff broadcast 218.185.71.31
inet 192.168.216.31 netmask 0xffffffff broadcast 192.168.216.31
media: Ethernet 10Gbase-T <full-duplex>
status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
groups: lo
lo1: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
inet 127.0.31.1 netmask 0xffffffff
groups: lo
lo2: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
groups: lo
search harte-lyne.ca
nameserver 216.185.71.33
nameserver 216.185.71.34
nameserver 127.0.0.1
options edns0
# $FreeBSD: releng/11.1/etc/hosts 109997 2003-01-28 21:29:23Z dbaker $
#
# Host Database
. . .
#
#
::1 localhost localhost.harte-lyne.ca
127.0.0.1 localhost localhost.harte-lyne.ca
The package management tool is not yet installed on your system.
Do you want to fetch and install it now? [y/N]: y
Bootstrapping pkg from
pkg+http://pkg.FreeBSD.org/FreeBSD:11:amd64/quarterly, please wait...
pkg: Error fetching
http://pkg.FreeBSD.org/FreeBSD:11:amd64/quarterly/Latest/pkg.txz: No
address record
A pre-built version of pkg could not be found for your system.
'ports-mgmt/pkg'.
PING 216.185.71.1 (216.185.71.1): 56 data bytes
^C
--- 216.185.71.1 ping statistics ---
5 packets transmitted, 0 packets received, 100.0% packet loss
Why does this jail not have a network connection?

After a quick check I guess you are missing NAT on the host (pf or ipfw)?

--
Herbert

James B. Byrne via freebsd-questions

2018-05-31 13:15:11 UTC

Permalink

On Wed, 30 May 2018 17:24:03 +0200, "James B. Byrne via

After a quick check I guess you are missing NAT on the host (pf or
ipfw)?

One does not require NAT when one has a public IP address assigned to
the I/F. The jail network traffic is not getting off the host system
as I have determined from tcpdump.

I have set jails up before on a similarly configured host and have
gotten them to work, albeit always with some difficulty or other.
--
*** e-Mail is NOT a SECURE channel ***
Do NOT transmit sensitive data via e-Mail
Do NOT open attachments nor follow links sent by e-Mail

James B. Byrne mailto:***@Harte-Lyne.ca
Harte & Lyne Limited http://www.harte-lyne.ca
9 Brockley Drive vox: +1 905 561 1241
Hamilton, Ontario fax: +1 905 561 0757
Canada L8E 3C3

Arthur Chance

2018-05-31 13:40:53 UTC

Permalink

Post by James B. Byrne via freebsd-questions

On Wed, 30 May 2018 17:24:03 +0200, "James B. Byrne via

After a quick check I guess you are missing NAT on the host (pf or
ipfw)?

I've just taken another look at your original mail. I think the key
might be in this

Post by James B. Byrne via freebsd-questions
JID IP Address Hostname Path
1 127.0.31.1 mx31 /usr/jails/mx31

Note address ^^^^^

Post by James B. Byrne via freebsd-questions
vtnet0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST>
metric 0 mtu 1500
options=80028<VLAN_MTU,JUMBO_MTU,LINKSTATE>
ether 58:9c:fc:0e:cd:bb
hwaddr 58:9c:fc:0e:cd:bb
inet 218.185.71.31 netmask 0xffffffff broadcast 218.185.71.31
inet 192.168.216.31 netmask 0xffffffff broadcast 192.168.216.31
media: Ethernet 10Gbase-T <full-duplex>
status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
groups: lo
lo1: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
inet 127.0.31.1 netmask 0xffffffff
groups: lo
lo2: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
groups: lo

Whatever you think is happening this shows your jail is using a loopback
address on a loopback socket. Packets sent from loopback devices or from
loopback addresses are simply not going to get off the box.

--
An amusing coincidence: log2(58) = 5.858 (to 0.0003% accuracy).

James B. Byrne via freebsd-questions

2018-05-31 14:21:11 UTC

Permalink

Post by Arthur Chance
I've just taken another look at your original mail. I think the key
might be in this

Post by James B. Byrne via freebsd-questions
JID IP Address Hostname Path
1 127.0.31.1 mx31
/usr/jails/mx31

Note address ^^^^^

The command jls reports the loopback address for all of the jails I
have defined on other hosts. For example:

[***@vhost02 ~]# jls
JID IP Address Hostname Path
2 127.0.34.1 hlldns04 /usr/jails/hlldns04
3 127.0.150.1 hllmx150 /usr/jails/hllmx150

[***@vhost02 ~]# ezjail-admin console hlldns04
Last login: Thu May 31 10:14:37 on pts/0
. . .
[***@hlldns04 ~]# pkg upgrade
Updating FreeBSD repository catalogue...
FreeBSD repository is up to date.
All repositories are up to date.
New version of pkg detected; it needs to be installed first.
The following 1 package(s) will be affected (of 0 checked):

Installed packages to be UPGRADED:
pkg: 1.10.3_1 -> 1.10.5

Number of packages to be upgraded: 1

3 MiB to be downloaded.

Proceed with this action? [y/N]:

This jail has no problem reaching the internet.
--
*** e-Mail is NOT a SECURE channel ***
Do NOT transmit sensitive data via e-Mail
Do NOT open attachments nor follow links sent by e-Mail

James B. Byrne mailto:***@Harte-Lyne.ca
Harte & Lyne Limited http://www.harte-lyne.ca
9 Brockley Drive vox: +1 905 561 1241
Hamilton, Ontario fax: +1 905 561 0757
Canada L8E 3C3

Arthur Chance

2018-05-31 14:29:37 UTC

Permalink

Post by James B. Byrne via freebsd-questions

Post by Arthur Chance
I've just taken another look at your original mail. I think the key
might be in this

Post by James B. Byrne via freebsd-questions
JID IP Address Hostname Path
1 127.0.31.1 mx31
/usr/jails/mx31

Note address ^^^^^

The command jls reports the loopback address for all of the jails I
JID IP Address Hostname Path
2 127.0.34.1 hlldns04 /usr/jails/hlldns04
3 127.0.150.1 hllmx150 /usr/jails/hllmx150
Last login: Thu May 31 10:14:37 on pts/0
. . .
Updating FreeBSD repository catalogue...
FreeBSD repository is up to date.
All repositories are up to date.
New version of pkg detected; it needs to be installed first.
pkg: 1.10.3_1 -> 1.10.5
Number of packages to be upgraded: 1
3 MiB to be downloaded.
This jail has no problem reaching the internet.

Addresses in 127/8 must not appear on the network anywhere
(https://tools.ietf.org/html/rfc5735#page-3), and FreeBSD has specific
checks in the networking code to prevent this. If any jail with such an
address is contacting the network then there must be some form of NAT
involved. I can only suggest you check for differences between the jails
that can get out and the one that can't *and* look for NAT on the
host(s) with jails that can get out.

--
An amusing coincidence: log2(58) = 5.858 (to 0.0003% accuracy).

James B. Byrne via freebsd-questions

2018-05-31 19:01:59 UTC

Permalink

Post by Arthur Chance

Post by James B. Byrne via freebsd-questions

Post by Arthur Chance
I've just taken another look at your original mail. I think the key
might be in this

Post by James B. Byrne via freebsd-questions
JID IP Address Hostname Path
1 127.0.31.1 mx31
/usr/jails/mx31

Note address ^^^^^

The command jls reports the loopback address for all of the jails I
JID IP Address Hostname Path
2 127.0.34.1 hlldns04 /usr/jails/hlldns04
3 127.0.150.1 hllmx150 /usr/jails/hllmx150

The 127.0.x.1 addresses are used by the cloned loopback interfaces
that the jails require. Traffic on those addresses is going nowhere
but back to the jail that owns them.

I have several hosts with multiple jails and on every one of them the
jls command displays the loopback address assigned to the jail.

[***@vhost04 ~ (master #)]# jls
JID IP Address Hostname Path
1 127.0.124.1 hll124 /usr/jails/hll124

[***@vhost02 ~]# jls
JID IP Address Hostname Path
1 127.0.150.1 hllmx150 /usr/jails/hllmx150
2 127.0.34.1 hlldns04 /usr/jails/hlldns04

[***@vhost03 ~]# jls
JID IP Address Hostname Path
1 127.0.151.1 hllmx04 /usr/jails/hllmx04
2 127.0.33.1 hlldns02 /usr/jails/hlldns02

I can go on but I believe that the point is made. Each of these jails
can reach the internet. Some hosts are on the same LAN segment as the
host with the jail I am having problems with. NAT is not involved as
the IP address assigned to the jail's virtual interface is public.

I have discovered my error. It is a typo in the IP address assigned
to the jail. I wrote 218.185.71.31 when it should have been
216.185.71.31.
I must have looked at that line in the jail configuration file a dozen
times or more and missed it.
--
*** e-Mail is NOT a SECURE channel ***
Do NOT transmit sensitive data via e-Mail
Do NOT open attachments nor follow links sent by e-Mail

James B. Byrne mailto:***@Harte-Lyne.ca
Harte & Lyne Limited http://www.harte-lyne.ca
9 Brockley Drive vox: +1 905 561 1241
Hamilton, Ontario fax: +1 905 561 0757
Canada L8E 3C3

Herbert J. Skuhra

2018-05-31 14:32:28 UTC

Permalink

Post by James B. Byrne via freebsd-questions

On Wed, 30 May 2018 17:24:03 +0200, "James B. Byrne via

After a quick check I guess you are missing NAT on the host (pf or
ipfw)?

OK, you are obviously smarter than me (or a magician). I'll continue
using NAT in such a setup. Adding the follwing to /etc/pf.conf works for me:

nat on $ext_if inet from 127.0.32.1/32 to any -> $ext_ip

--
Herbert