Online play behind pfSense not possible

[pixel24]

Hi@all,

At home, I "sit" behind an ordinary router and have no problems joining or hosting a game in the online lobby. The router does NAT and I have no port forwarding set up. The client PC is always the same. Ubuntu 20.04 with the firewall disabled.

In the office I have a pfSense firewall host. It does not work there. It tries to connect for a long time and in the end it says it doesn't work.

Does anyone have experience with how to configure the pfSense so that 0 AD behind it works?

with best

pixel24

Edited September 27, 2021 by pixel24

[Stan`]

Hi Pixel24,

Did you try to open port 20595 ?

 

[dave_k]

Hey pixel24,

Are you the owner of the workplace and/or allowed to play games while at work? And, are you the administrator of the pfSense router and have access to the administration interface or can instruct the network administrator to change settings?

If so, then I recommend that the network administrator try this advice in the pfSense software manual in order to support online games.

https://docs.netgate.com/pfsense/en/latest/nat/compatibility.html#online-games

If these tips don't solve the problem then join IRC (click "IRC Channel" at the top of the 0ad website) during American daylight hours, and I'll help you with troubleshooting while you're at the workplace.

[pixel24]

Yes, I am the owner / administrator and may play :-)

I have read the Netgate documentary. It says I should enter it under NAT -> Outgoing.

Does the port 20595 incoming or outgoing have to be open? UDP, TCP or both?

I have tried it on the pfSense under Firewall / NAT / Outgoing:

Unfortunately without success.

Edited September 28, 2021 by pixel24

[pixel24]

not work

Edited September 28, 2021 by pixel24

[pixel24]

OK, I spoke too soon. Once I was able to join a game as a spectator. Now it no longer works

[pixel24]

I got it right. You have to work on the pfSense:

Firewall / NAT / Outgoing

Enable hybrid creation of NAT rules and set the mapping for port 20595 (static). See picture

[pixel24]

I haven't played in the office yet. Unfortunately, I didn't have time. I have tried from time to time to join a game as a spectator via the lobby.

It was strange that it often works but sometimes hangs when I try to connect to .... and then the error with the firewall and port comes up again.

I will investigate and test this further and report back.

If anyone else has an idea for me. Gladly :-)

 

[dave_k]

Intermittent results when joining other players' games are not necessarily an indication that you have things configured incorrectly at your end. There are many players who attempt to host, but who have flaws in their router connections or configurations. Some clients are able to connect to them seemingly reliably, while other clients are not able to connect to them at all. Or, perhaps clients are intermittently able to connect to them. One example of a potential cause of these problems is a double-NAT configuration.

If the problem is caused by the configuration of your router then you need to do more thorough testing. You need to see whether you are actually maintaining a long-term connection to games you try to spectate. It is possible that your router is only briefly opening the necessary ports and closing them within a minute, or so. Why would it do this? Perhaps it is using port triggering instead of port forwarding. Or, perhaps it has rate limiting applied to new UDP connections.

Consider using Wireshark or tcpdump to capture network traffic at various places in order to help diagnose where and why problems are occurring. Start by capturing traffic of proper functionality so that you have a baseline with which to compare captures of traffic during improper functionality. And, consider using an alternative router at your workplace temporarily in order to rule out other possible causes of the problem, such as ISP filtering, damaged network cables, software-based firewalls, or problematic hardware (e.g. an Intel Puma chipset in the modem).

I think that it's better to do testing among players who have known correct network configurations and who do not have network hardware with the Intel Puma chipset (which would cause intermittent packet loss). I am one such player, and I'm able to host reliably. If you come to the #0ad IRC channel on irc.quakenet.org (click the "IRC Channel" link at the top of the page) during American daylight hours then I'll help you with troubleshooting.

By the way, what was the result of trying the advice in the pfSense manual that I linked earlier? For example, the "Static Port" setting?

Edited September 30, 2021 by dave_k

[pixel24]

3 hours ago, dave_k said:

Intermittent results when joining other players' games are not necessarily an indication that you have things configured incorrectly at your end. There are many players who attempt to host, but who have flaws in their router connections or configurations. Some clients are able to connect to them seemingly reliably, while other clients are not able to connect to them at all. Or, perhaps clients are intermittently able to connect to them. One example of a potential cause of these problems is a double-NAT configuration.

I already had this suspicion too.

3 hours ago, dave_k said:

If the problem is caused by the configuration of your router then you need to do more thorough testing. You need to see whether you are actually maintaining a long-term connection to games you try to spectate. It is possible that your router is only briefly opening the necessary ports and closing them within a minute, or so. Why would it do this? Perhaps it is using port triggering instead of port forwarding. Or, perhaps it has rate limiting applied to new UDP connections.

Consider using Wireshark or tcpdump to capture network traffic at various places in order to help diagnose where and why problems are occurring. Start by capturing traffic of proper functionality so that you have a baseline with which to compare captures of traffic during improper functionality. And, consider using an alternative router at your workplace temporarily in order to rule out other possible causes of the problem, such as ISP filtering, damaged network cables, software-based firewalls, or problematic hardware (e.g. an Intel Puma chipset in the modem).

Bypassing the firewall here would only be possible with a great deal of effort. But I can (almost) rule out the hardware. The pfSense runs as a VM on a Proxmox server. The hardware is from Supermicro (server). The cabling is new and has been measured. We have also made various throughput measurements and checked for possible packet losses.

 

3 hours ago, dave_k said:

I think that it's better to do testing among players who have known correct network configurations and who do not have network hardware with the Intel Puma chipset (which would cause intermittent packet loss). I am one such player, and I'm able to host reliably. If you come to the #0ad IRC channel on irc.quakenet.org (click the "IRC Channel" link at the top of the page) during American daylight hours then I'll help you with troubleshooting.

I will test a longer game here with a friend to see if it works stably. I would also be happy to take you up on your offer.

 

3 hours ago, dave_k said:

By the way, what was the result of trying the advice in the pfSense manual that I linked earlier? For example, the "Static Port" setting?

This was helpful and led to the fact that it works at all. Static Port is mandatory for the pfSense for this purpose.

[dave_k]

Hey pixel24,

Can we get an update on whether 0ad is working at your workplace? If it's still not working properly then I have some more advice. Choose which art of troubleshooting you want: Black Art or White Art. Both can be useful, I think.

 

Black Art method of troubleshooting (recipes that might or might not solve the problem magically)

1. Advice on getting UDP gaming to work with pfSense is here. This mentions VoIP networking. VoIP usually uses UDP protocol, which is also true of most online games, including 0ad.

2. You have configured a 1:1 NAT port mapping. Do you know what 1:1 NAT is? Is that actually how you want to configure your network? Most consumer routers are configured for cone NAT instead of 1:1 NAT. But, maybe 1:1 NAT is correct for your workplace ISP.

3. Here is an example of something besides a firewall that can block traffic. This has affected me, even though I'm not using an optical network, despite what Intel's release notes state for the Windows fix.

Instructions on disabling RX checksum hardware offloading in order to workaround bugs in Realtek and Intel network adapter hardware
Instructions for Windows (this might be necessary if you are using Windows as the host of the VM for pfSense):
 

Instructions for Linux (this might be necessary if you are using Linux as the host of a VM for pfSense):
1. At a minimum, add the following line to the iface section in /etc/network/interfaces or /etc/systemd/network/* for each Intel Gigabit or 10Gigabit network adapter.
       offload-rx off
2. If necessary, add the following line, as well.
       offload-tx off
3. For testing on the command line, use the following commands, replacing "[iface]" with the network adapter interface name, such as eth0 or enp1s0.
       /sbin/ethtool --offload [iface] rx off
       /sbin/ethtool --offload [iface] tx off
       # Note that the changes made with ethtool will be lost after rebooting the computer.
       # It is better to permanently configure the changes in your OS-managed network configuration

4. Another problem is packet loss caused by the Intel Puma chipset in routers and modems. Read the info here and here.

Their list of affected modems and routers is incomplete. To check whether your modem or router is affected, determine which chipset it uses. https://deviwiki.com can help with this if the network device is not relatively new.

There does not seem to be a fix for it other than to replace the hardware. Intel has attempted to provide firmware updates, but they have only swept the problem under the rug by making pings reliable and other traffic still unreliable. There is a class action lawsuit about this.

 

White Art method of troubleshooting

The White Art method involves the following: read the manuals; learn more about how things work; learn how to diagnose where the problem is occuring and what the cause(s) are through controlled experimentation and examination of logs and configuration; and learn how to fix it properly)

First, you should setup a "development" router or VM image that is separate from your "production" router or VM image. This way you do not end up breaking your workplace router system while learning how things work and testing configuration changes.

I was troubleshooting a Linux-based router recently and realized that it's useful to see a log of blocked traffic or statistics of packets blocked by each firewall rule. The pfSense documentation has advice on this here.

All pfSense documentation -- explore and learn!

The reason that this is important is because you need to verify whether the firewall is even the cause of the networking problem. To do this you can check the firewall logs for evidence of 0ad traffic being blocked.

Also, a simpler test is to play a game of 0ad without using the lobby. Ask a friend to host outside the lobby and tell you his IP and port. It's also the most likely to be blocked by the router in some way.

Here are questions that I would ask if I were in a real-time chat with you. By the way, is there a convenient means of communication that you would prefer instead of IRC? I haven't seen you on IRC. Would it help to use something supported by your mobile phone like Discord, Skype, or a phone call within the US?

1. Where is your modem?
2. Where is your router? (Maybe the pfSense system is your router, but it's probably not also your modem)
3. Where is your managed switch (if you have one)?
4. What are the brand and model of each?
5. How are they configured? Are they doing NAT or bridging? Are they doing firewalling as well?
6. What are the logs indicating at each device?
7. Have you tried a simpler test, such as treating the pfSense router as a Device Under Test (DUT)? I mean, disconnect the modem (and therefore the Internet) from the pfSense router and connect a spare computer to the WAN port of the pfSense router in order to mimic an Internet-based host for a game of 0ad. See whether this affects the symptoms, or not. Then vary one variable at a time for controlled experimentation.

Don't just focus on the pfSense system; look at the other elements of the network. But, also focus on the system behind the pfSense system. The host hardware and operating system have configurations that can affect 0ad traffic.

To list statistics for firewall rules, use pfInfo or query raw rules with the command line (see below for instructions on querying raw rules with the pf command).

The more that you know about how your networking system works, the more capable you will be of troubleshooting it. You can look under the hood of pfSense. It is based on FreeBSD. It allows SSHing in for a command line. Then you can list the raw firewall rules (pf rules).

Here is the documentation for the pf packet filtering system. The most important types of statements in pf.conf are "Packet Filtering" and "Translation".

To query the raw rules of the firewall from an SSH command line, you would likely use these commands.

# list the currently loaded filter rules with per-rule statistics
pf -s rules -v

# list the currently loaded NAT rules with per-rule statistics
pf -s nat -v

# show per-rule statistics, including packets and bytes affected by that rule
pf -s labels -v

Making sense of this involves knowing how the pf firewall system works. Here is some relevant documentation.

I would advise resisting the urge to manually override aspects of the firewall configuration with pf commands. If you don't understand enough of the system to fix it with the pfSense front-end then changing raw rules is not going to be a solution --- at least, not without also opening up too much of the firewall.

More pf commands are explained here.

Once you have read some of the documentation, you will be able to setup a test FreeBSD system (in a separate VM or ideally on bare metal hardware) with an experimental basic firewall configuration. Then you can triangulate between pfSense and a progressively more complex test FreeBSD system in order to figure out where exactly the firewall is blocking 0ad traffic (if it's even caused by the firewall).

FreeBSD is a free download here. Click the correct architecture under "Installer images" to find download links.

Documentation on setting up FreeBSD is here.

pfSense offers professional support via email (called TAC PRO) for $399/year. They offer professional support via telephone (called TAC ENTERPRISE) for $799/year. But maybe you have an employee who knows the system very well and can troubleshoot this easily.
https://www.netgate.com/support

If the firewall has no indication in the logs that it is blocking 0ad traffic then the traffic is either being blocked by a rule that doesn't make a log entry, or the traffic is being blocked by something else, such as the RX TCP/IP-v6 checksum offloading bug of certain network adapters.

[pixel24]

Please excuse the late reply. There is a lot going on in the office.

On 05/10/2021 at 9:10 PM, dave_k said:

Can we get an update on whether 0ad is working at your workplace? If it's still not working properly then I have some more advice. Choose which art of troubleshooting you want: Black Art or White Art. Both can be useful, I think.

Yes, it (almost) always works. I have read that if it does not work, it may be due to the remote station. That is, the one that hosts the game.

On 05/10/2021 at 9:10 PM, dave_k said:

1. Advice on getting UDP gaming to work with pfSense is here. This mentions VoIP networking. VoIP usually uses UDP protocol, which is also true of most online games, including 0ad.

I have deactivated the rewrite for UDP port 20595.

On 05/10/2021 at 9:10 PM, dave_k said:

2. You have configured a 1:1 NAT port mapping. Do you know what 1:1 NAT is? Is that actually how you want to configure your network? Most consumer routers are configured for cone NAT instead of 1:1 NAT. But, maybe 1:1 NAT is correct for your workplace ISP.

No, I don't know the difference between 1:1 and cone NAT in detail. Where did I configure that?
I have set up the pfSense according to the instructions and help in the Netgate forum.
The WAN interface of the pfSense is connected to a media converter (RJ45 <-> glass fibre). I don't know if this is relevant.

On 05/10/2021 at 9:10 PM, dave_k said:

3. Here is an example of something besides a firewall that can block traffic. This has affected me, even though I'm not using an optical network, despite what Intel's release notes state for the Windows fix.

Instructions on disabling RX checksum hardware offloading in order to workaround bugs in Realtek and Intel network adapter hardware
Instructions for Windows (this might be necessary if you are using Windows as the host of the VM for pfSense):

Yes, I know that. I already deactivated this during the installation.

 

On 05/10/2021 at 9:10 PM, dave_k said:

The White Art method involves the following: read the manuals; learn more about how things work; learn how to diagnose where the problem is occuring and what the cause(s) are through controlled experimentation and examination of logs and configuration; and learn how to fix it properly)

First, you should setup a "development" router or VM image that is separate from your "production" router or VM image. This way you do not end up breaking your workplace router system while learning how things work and testing configuration changes.

I was troubleshooting a Linux-based router recently and realized that it's useful to see a log of blocked traffic or statistics of packets blocked by each firewall rule. The pfSense documentation has advice on this here.

All pfSense documentation -- explore and learn!

The reason that this is important is because you need to verify whether the firewall is even the cause of the networking problem. To do this you can check the firewall logs for evidence of 0ad traffic being blocked.

Also, a simpler test is to play a game of 0ad without using the lobby. Ask a friend to host outside the lobby and tell you his IP and port. It's also the most likely to be blocked by the router in some way.

Yes, I know I should do that. But I can only do it piecemeal because I have to work a lot.

 

On 05/10/2021 at 9:10 PM, dave_k said:

Here are questions that I would ask if I were in a real-time chat with you. By the way, is there a convenient means of communication that you would prefer instead of IRC? I haven't seen you on IRC. Would it help to use something supported by your mobile phone like Discord, Skype, or a phone call within the US?

Unfortunately, my English is not (yet) good enough for real-time communication. I always need a translator to help me :-(

I will find out the answers to the questions when I am in the office (home office at the moment).

with best

pixel

[dave_k]

15 hours ago, pixel24 said:

Yes, it (almost) always works. I have read that if it does not work, it may be due to the remote station. That is, the one that hosts the game.

Okay, this is good to hear. Does it always work when you connect to players who are known to have properly configured their networks? Are you able to host games? On the other hand, "if it isn't broken, don't fix it," can be a useful aphorism.

15 hours ago, pixel24 said:

No, I don't know the difference between 1:1 and cone NAT in detail. Where did I configure that?

See the screenshot of the router administration interface for "port weiterleitung" (port forwarding), which you posted in this thread in your last post on September 28. When I first looked at the screenshot I made an assumption that 1:1 port mapping was enabled, but maybe it's not. There is no red line below it, so maybe it's just an alternate option that is not currently enabled.

I have translated the German interface using Google Translate (which was able to infer the correct accents on characters). For anyone else reading this, here is the translation.

Die anderungen wurden erfolgreich angewandt. Das Firewallregelwerk wird nun im Hintergrund neu geladen.
Uberprufen Sie den Fortschritt des Filterneuladens.
Port Weiterleitung
Ausgehend
NPt
Ausgehender NAT Modus
Automatische Erzeugung der NAT-Regel fur den ausgehenden Verkehr. (IPSec Durchleitung inklusive)
Hybride Erzeugung der NAT-Regeln. (Automatische ausgehende NAT & folgende Regeln)
Manuelle Erzeugung der Regeln fur ausgehendes NAT. (AON - Advanced Outbound NAT)
Automatische Generierung von ausgehenden NAT regeln deaktivieren. (Keine ausgehenden NAT Regeln)
Schnittstelle
Quelle
Quellport
Ziel
Zielport
NAT-Adresse
NAT-Port
Statischer Port
Beschreibung
Aktionen

The changes were successfully applied. The firewall regulator will now be reloaded in the background.
Check the progress of the filtering backup.
Port forwarding
Starting
NPt
Outgoing NAT mode
Automatic generation of the NAT rule for outgoing traffic. (IPSec throughput inclusive)
Hybrid generation of the NAT rules. (Automatic outgoing NAT & following rules)
Manual generation of the rules for outgoing NAT. (AON - Advanced Outbound NAT)
Disable automatic generation of outgoing NAT rules. (No outgoing NAT rules)
Interface
Source
Source port
Target
Destination sport
NAT
NAT-PORT
Static port
Description
Actions

It helps to know the difference between source NAT and destination NAT. I think that source NAT is designed for connecting as a client to other players' hosted games. And, destination NAT is designed for allowing other players to connect to your hosted game. Destination NAT is also known as port forwarding. Destination NAT is usually only necessary if you want to be able to host games. However, maybe it is also necessary if you want to enable static port mapping.

I am curious what the effects are of each of the modes for "Ausgehender NAT Modus" (Outgoing NAT mode). Capturing traffic and listing the raw rules with each of the choices for the "Ausgehender NAT Modus" could help answer the question. I would advise that you avoid posting your raw firewall rules on a public forum, though, as it could show untrustworthy people gaps in your firewall configuration.

Here is some info on the difference between source NAT and destination NAT (in particular, see chapter 3, "The Two Types of NAT", and chapters 6.1 and 6.2). Yes, it's talking about Linux and netfilter, but the explanations of networking concepts are universal for any operating system. And, it might be useful for internet searching to find more detailed explanations.

If things work well enough then experimentation and troubleshooting can be a long-term project, of course. Let me know if you have any other questions.