gmane.os.openbsd.pf

Re: Simultaneous CARP failover for multiple interfaces

Kyle Lanclos — 2012-04-27T18:05:47

I quoted from the OpenBSD FAQ:

I re-did my tests this morning, and yes indeed, both CARP interfaces in
the CARP group switch automatically between the BACKUP and MASTER states
when I do abusive things to one of the connections. I say "automatically"
instead of "simultaneously" because there was a bit of a delay while the
reconnected link negotiated speed. I blame Cisco for that.

Thanks for all the assistance, I found this exchange to be quite helpful.

--Kyle

Re: Simultaneous CARP failover for multiple interfaces

Daniel Hartmeier — 2012-04-24T06:57:36


Without any active probing, like with ifstated, there is no way to
distinguish which uplink is "up but not forwarding". It could be either
the master's, or the backup's, or both. Statistically, for every time you
improve the situation by failing over, there is a time you shoot yourself
in the foot doing the same. If you do nothing, you have the same chances,
and things remain simpler.

With ifstated, you only need to change one side's advskew so their order
reverses, then rely on carp's election process. For instance, run
ifstated only on the master, pinging next hops on all sides, and

 - when any ping fails (the first time), demote:
     increase own advskew above the backup's

 - when all pings succeed (again), promote:
     reset own advskew to the original value (below the backup's)

In your example above, ifstated on the master would detect a ping
failure on one next hop and demote by increasing its advskew above the
backup's.

With preempt enabled, the master would lose election on the other
interface and therefore group failover all interfaces to backup state,
while the backup would win election and group failback all interface to
master state, i.e. self-promotion of the backup is done only through carp
election.

When you fix the interface, ifstated will see all pings succeed again,
and reset advskew. Now the (preferred) master wins election and fails
back.

I don't think there is a case where it's helpful to run scripts on both
the master and the backup. You'd have to be careful to not introduce new
failure cases, for instance when a next hop is unreachable from both.

Daniel

Re: Simultaneous CARP failover for multiple interfaces

Kyle Lanclos — 2012-04-23T21:34:31

I originally wrote:

It sounds like using 'defer' may allow pf + pfsync to handle the issues
resulting from asymmetric routing of packets, as long as the asymmetry
is fully contained within the pfsync'd hosts.

I apologize if I gave too much airtime to the pf + pfsync aspects of
what I was trying to resolve, we largely worked around those by enabling
carp preemption.

--Kyle

Re: Simultaneous CARP failover for multiple interfaces

Kyle Lanclos — 2012-04-23T21:23:20

Whoops, thanks for pointing that out. I will re-do my tests (the setup is
at a remote site), and make sure that I'm done making things up.

I'm not sure where your quote came from; I see similar text in NetBSD docs,
but this is what I (now) find in the OpenBSD FAQ:

net.inet.carp.preemptAllow hosts within a redundancy group
that have a better advbase and advskew
to preempt the master. In addition, this
option also enables failing over a group
of interfaces together in the event that
one interface goes down. If one physical
CARP-enabled interface goes down, CARP
will increase the demotion counter,
carpdemote, by 1 on interface groups that
the carp(4) interface is a member of, in
effect causing all group members to fail-
over together. 

http://www.openbsd.org/faq/pf/carp.html

I did establish an interface group for my two CARP interfaces, but I did
not do my failover tests while it was in that state. As I said, I clearly
need to re-do my tests.


At least for our case, a switch failure on either side of the firewall/router
represents a total loss of connectivity.

However, this does jog another potential failure mode. Some of our older
OpenBSD firewalls (going back to OpenBSD) will occasionally (maybe once a
year) "lose" a network interface. If you logged in at the console of a
host while it was in this state, the interface would look perfectly normal,
but it would not pass any traffic. I callously worked around this by
administratively cycling each network interface on the affected machine(s)
on a weekly basis.

If we ran into this failure mode with our CARP firewalls, I'm assuming the
master would keep right on thinking it was the master, and not attempt to
demote iteslf.

While it is certainly helpful for self-demotion of a master to occur,
it seems reasonable for self-promotion of a slave to also occur.


Thanks, I'll look into that.

--Kyle

Re: Simultaneous CARP failover for multiple interfaces

Karl O. Pinc — 2012-04-23T21:20:56


I didn't notice _any_ reference to pfsync in the original
post.  Perhaps this is part of the problem?



Karl <kop< at >meme.com>
Free Software:  "You don't pay back, you pay forward."
                 -- Robert A. Heinlein

Re: Simultaneous CARP failover for multiple interfaces

Stuart Henderson — 2012-04-23T20:19:44

pfsync(4)'s "defer" option might help. there is a penalty but it might
be acceptable for your use case.

     Where more than one firewall might actively handle packets, e.g. with
     certain ospfd(8), bgpd(8) or carp(4) configurations, it is beneficial to
     defer transmission of the initial packet of a connection.  The pfsync
     state insert message is sent immediately; the packet is queued until
     either this message is acknowledged by another system, or a timeout has
     expired.  This behaviour is enabled with the defer parameter to
     ifconfig(8).

Re: Simultaneous CARP failover for multiple interfaces

Daniel Hartmeier — 2012-04-23T20:56:32


Yes, it will:

     net.inet.carp.preempt       Allow virtual hosts to preempt each other.
                                 It is also used to failover carp interfaces
                                 as a group.  When the option is enabled and
                                 one of the carp enabled physical interfaces
                                 goes down, advskew is changed to 240 on all
                                 carp interfaces.  See also the first example.
                                 Disabled by default.

(i.e. this single sysctl knob enables both group failover and failback)

This covers link state change (unplugged cable) as well as
administrative down of the physical interface.

It does not cover the case where the link remains up, but the uplink
switch stops forwarding, for instance, but...


.. see ifstated(8), which can ping uplink hops and issue ifconfig advskew
changes to demote the master when appropriate.

Daniel

Simultaneous CARP failover for multiple interfaces

Kyle Lanclos — 2012-04-23T18:49:14

I have a pair of OpenBSD firewall/routers in a reasonably vanilla
pf + pfsync + CARP configuration, each straddling two routed networks.
The CARP interface on the internal network is the default gateway for
that subnet. The CARP interface on the external network is the default
destination for traffic aimed at the internal network.

It all works splendidly, with one exception.

In order for our firewall to operate effectively, we use 'keep state'
pf rules. We empirically determined that we must have CARP preemption
enabled, otherwise pf cannot properly establish state for new TCP
connections. If pfsync could be told to synchronize incomplete states,
this issue might go away.

Example: firewall1 is the master on the carp1 interface, and firewall2
is the master on the carp2 interface. Inbound traffic to an internal
host arrives via the carp1 interface, and return traffic arrives via
the carp2 interface. pf will not establish state for this new connection
since the inbound and return traffic are not handled by the same firewall
host.

We thus use CARP preemption to force one of the firewalls to always come
up as the master for both CARP interfaces. This is not so unresaonable,
though it might be nice if the documentation presented this use-case (or
similar) as a rationale for needing CARP preemption.

Where this presents a problem is if the current CARP master loses a single
network interface (cable unplugged, isolated hardware failure, sysadmin
failure, etc.), as opposed to the CARP master failing entirely. The slave
will appropriately assume the master role for one CARP interface, but will
*not* do so for the second.

Beyond the basic asynchronous routing + state creation issue described
above, this specific failure mode would still result in a complete inability
to pass traffic in a single direction, since packets would come into the
half-dead master via its good interface, but could not be forwarded to the
network associated with the failed interface.

We would like our otherwise nicely redundant firewall configuration to be
resilient against this type of failure. Short of running a cron job every
sixty seconds to check the interface state, is there some way we can
automatically force the promotion of a CARP slave if a second CARP interface
flips from slave to master?

Suggestions are most appreciated. I apologize if the CARPish-flavor of this
question is not entirely appropriate for the pf mailing list; if there is
another list that would be more suited for this question, please let me know.

--Kyle

Re: inbound queueing on external interface due to multiple internal interfaces

Karl O. Pinc — 2012-04-12T14:38:01

I am all wet then.  Isn't the first time.  Sorry to distract you.



Karl <kop< at >meme.com>
Free Software:  "You don't pay back, you pay forward."
                 -- Robert A. Heinlein

Re: inbound queueing on external interface due to multiple internal interfaces

Kelley Reynolds — 2012-04-12T12:56:35


I don't know if your setup will support it but you could make a trunk interface with queues then use VLANs instead of physical interfaces, works fine.

Kelley Reynolds
CTO / Cofounder, RubyScale

Re: inbound queueing on external interface due to multiple internal interfaces

Stuart Henderson — 2012-04-12T12:40:47

Yes, that thread is talking about creating multiple queues with the
same name, this is good for traffic classification but it is not a
single queue shared between multiple interfaces.


Currently two machines is the cleanest (and maybe only) way to do this,
I have a feeling you'll get better results making these route rather than
bridge.

altq's queues are all per-interface. Anything to change this would
be major work. Plan is for the queue code to be completely replaced
so working on this before that happens would get in the way.


lo(4) is a bit of a specialist interface to be trying this with.

You might get somewhere with vether(4), the uncommitted diff "virtual
crossover link with vether(4)" on tech< at >openbsd.org could be of interest.

Another option might be to route things to a userland daemon via tun(4)
that reinjects the packets then you can queue on the tun interface
instead. I have some old code from kjc sitting in my inbox that might
be useful here, I'll try and find time to dust it off. Higher overhead
but somewhere that is going to need to 'borrow' downstream bandwidth
between interfaces is likely to not be seeing super-high-speed
traffic anyway so this might not be a problem.

Re: inbound queueing on external interface due to multiple internal interfaces

Andy Lemin — 2012-04-12T09:48:52

Hi, thanks very much for your suggestion.


I read through that archive but the problem it seems to be discussing seems=
 pretty trivial as its discussing semantics of queuing downstream traffic w=
hich you just do by queuing on the internal interface (which works perfectl=
y if you only have one internal interface).

It does suggest creating a queue structure without assigning a specific int=
erface as you suggest, however when you do this I believe PF simply creates=
 copy's of the queue structure and creates instances per interfaces? I.e. t=
he different interfaces are not sharing the same physical queue in memory. =
Is this true?

I need to be able to perform downstream queuing pre-routing effectively so =
that all the downstream traffic (regardless of the internal zone) is all qu=
eued equally as one, allowing all internal interfaces to share the total do=
wnstream bandwidth.



Sadly we don't have enough ports/slots on our firewall to add a physical lo=
op back and so it has to be a virtual interface :(



Can you tell me why you feel it's a hack to do it using a virtual interface=
 (assuming it works)?
It seems like a legitimate solution for queuing downstream traffic pre-rout=
ing (altq processing is done during the last stage of egress processing) as=
 this would allow you to define unique downstream queues on any/multiple so=
urces (WAN or VPN) etc.

Or do you think it would be better to allow multiple internal interfaces to=
 share the same physical queue in memory?

Whilst its an ALTQ thing, I would assume both solutions would be best imple=
mented in PF?

I'm just looking for ideas to try as I'm genuinely surprised BSD can't do t=
his :(



Yes I'll keep my progress posted, as I know their are a lot of other people=
 out there who also want this functionality considering the number of threa=
ds I have come across on this, some dating back years.



Indeed ;)



That's a little out of my budget of a few hundred dollars! :(

Cheers for your time, Andy.

Re: inbound queueing on external interface due to multiple internal interfaces

Karl O. Pinc — 2012-04-11T16:48:46

You will want to see this thread:
Working example of bi-directional asymmetric ALTQ + NAT ruleset?
http://marc.info/?t=129472965800001&r=1&w=2

It talks about being able to have a single queue
on more than one interface, so you'd use a single
outbound queue on all your internal interfaces to
effectively rate-limit your inbound wan traffic.
You'll want to use the hfsc scheduler because
you're trading bandwidth for latency.  And hfsc has
sub-queues too so that might help allocate the
traffic per internal interface.  I haven't thought 
about this in quite some time but I think that this approach
will work.  But because I've not thought about
it I could be all wrong.  :-)

It's not a perfect solution; it won't work in the
general case where you've more than one interface
you want to limit inbound traffic on.

If sharing a queue on your internal 
interfaces does not do it you could get ugly and
use an extra 2 real interfaces (instead of the loopback interface
as you suggest) and a separate routing table and physically
loop the traffic back.  This is less ugly than having another box.

I suspect the loopback interface approach won't work, but that's
a total guess.  If it does work I'm not sure I'd want
to count on such a kludge continuing to work long-term.

I'm very interested in what works and what doesn't so
it would be good to hear back from you.


Don't forget the "empty" ack packets.


I heard the number $20,000 (US) thrown around.  I have no idea if
that's a realistic number.




Karl <kop< at >meme.com>
Free Software:  "You don't pay back, you pay forward."
                 -- Robert A. Heinlein

inbound queueing on external interface due to multiple internal interfaces

Andy Lemin — 2012-04-11T13:02:41

Hello,
I know this has been discussed before, but I am not a developer (I
wish I did have the skills as I would code this myself otherwise).
I have a huge respect for all that the OpenBSD community does and this
is not a winge or a moan, please hear our woes.


Yes, of course downstream queuing is only of benefit for TCP to get
the sender to slow down etc etc, but that is exactly what we need.
IF we don't queue downstream traffic, our downstream link gets
saturated and our ISP starts to drop random packets. If we do the
downstream queuing here, we pre-emptively drop low priority packets
before saturation thus slowing the sender down before saturating the
WAN's downstream leaving headroom for VoIP etc.


We have a firewall with 4 internal interfaces for various different
subnets and 1 external WAN interface like many users have. Queuing
upload traffic is easy on the WAN interface, but downstream traffic
needs to be queued on each of the internal interfaces.
Initially this would seem fine, until you realise that if 3 of the 4
internal interfaces are idle the busy 4th interface cannot use the
entire download bandwidth available on the WAN. This is because you
can only assign a 'slice' of the total download bandwidth (summing to
100% of WAN speed) to each of the internal interfaces resulting in
very inefficient download utilisation across the internal zones.

I appreciate that we could buy 'another' 2 boxes with only two
interfaces to sit on the WAN link (both running CARP etc) but this
seems extremely inefficient, wasteful, harder to manage and simply
ugly when compared to other features of OpenBSD, to end up with 4
boxes (2 multi-zone firewalls, and 2 queue servers) to achieve
resilient download queuing shared across multiple local subnets.

So, their is a clear need to be able to queue inbound traffic on the
WAN interface when there are multiple internal local interfaces. I
appreciate this is a feature that would need to be added to ALTQ than
PF.


So far my thinking is to create a second virtual loop-back interface
'lo1', and divert (divert-to not redirect (rdr-to)) all inbound
traffic on the WAN interface to lo1. Then set-up a queue on lo1 which
represents the desired shared 'inbound queue' for all internal
interfaces. Naturally after the traffic passes through lo1 it will get
queued, and the packets should then get routed as normal via their
respective internal local interfaces etc.

Would this work?

We have to use inbound queuing, without it our WAN link saturates with
low priority traffic, and we need to maintain headroom for high
priority VoIP traffic etc.
But we also want any internal local network to be able to fully
utilise the WAN downstream bandwidth instead of only 1/4 of it even
when the others are idle.


Thanks for your thoughts and ideas in advance :)
Andy

If we had to bounty this, how much? I might be able to get =A3100 for a
bounty?
Cheers.

CARP ip balancing on ExtremeWare

Kevin Bowling — 2012-02-07T10:03:36

I'm having a hell of a time using Extreme Networks Summit 400-24t
switches with IP balancing of any type.

I've tried OpenBSD 5.0 and a -current snapshot from Feb 02.  I've
tried all the modes, but none of them work.  There's not a good way
I'm aware of to do port mirroring for ip-unicast, but I don't
understand why ip-stealth isn't working.  I manually clear the
forwarding database after activating ip-stealth.

I'm just about to relegate these to dumb switch duty and try and find
some other vendor that just works.  Any chance someone else has
cracked the code on these with pf in the past?

Regards,
Kevin

Re: handling local traffic

Justin Murdock — 2012-01-30T13:23:47

I've been thinking about this for months (on and off) but as soon as I 
send the email a solution occurs to me. Typical.

On 30/01/2012 12:31, Justin Murdock wrote:
I can, however, catch this traffic using
     match out log user != unknown

similarly becomes:
     pass on $dmz to port {http, https, ssh}
     block in user != unknown
     pass in from <trusted> to port ssh user != unknown

I'm not entirely comfortable with this - especially as the "... in ... 
user != unknown" construction depends on there being a listening socket 
- which could be quite interesting with the rdr-to interactions.

handling local traffic

Justin Murdock — 2012-01-30T12:31:55

OpenBSD 4.9 GENERIC.MP#819 amd64

I'm not quite sure when things changed, but I can no longer apply rules 
to locally originating traffic:
     match in log on lo

now only logs local->local traffic and
     match out log received-on lo

logs nothing. The best I can do, it seems, is to
     match in tag "EXTERNAL"
     match out log tagged ""

More worrying for me, however, is the inability to control traffic being 
received by the host. I want to be able to

     pass on $dmz to port {http, https, ssh}
     block out on lo
     pass out on lo from <trusted> to port ssh

I feel I must be missing something, I'm just not sure what.

Re: Matter with transparent proxy

Stuart Henderson — 2012-01-12T15:52:37

There are quite clear instructions for transparent proxy configuration in the
pkg-readme file for the squid package (see /usr/local/share/doc/pkg-readmes).

Matter with transparent proxy

pizzahut — 2012-01-12T14:15:54

Hello,

I try now to create a transparent proxy using squid and using OpenBSD 5.0
Packet Filter all by passing a bridge.

The squid run I tested the bridge walking machines located on the other side
can access the outside.

The problem is that when I try to redirect traffic using packet filter to
127.0.0.1 on port squid listening (port 3128) nothing happens the machine is
on the other side can access the Internet and suffers no restrictions
previously configured in the squid.

If anyone has an idea of the problem please.

thank you

Re: I want copy pf.conf from FreeBSD 8.2 to OpenBSD 5 and use it

Karl O. Pinc — 2011-11-07T14:59:27

Have you looked at the pf faq at www.openbsd.org or the man page,
especially the "Translation" section?

I seem to recall simple nat examples that you could use
once you change the IP numbers/networks to the ones you're
using.

I don't understand your english but it sounds like you may
want "match binat-to" to get a 1-to-1 mapping between internal
and external IP numbers -- either assigned per individual ip
number, or with "bitmask" or perhaps with some other natting
method and "sticky-address".

Karl <kop< at >meme.com>
Free Software:  "You don't pay back, you pay forward."
                 -- Robert A. Heinlein

Re: I want copy pf.conf from FreeBSD 8.2 to OpenBSD 5 and use it

Gary — 2011-11-07T16:28:37


If all you want is one-to-one NAT and no packet inspection or filtering, why do you need a firewall at all? It's not good practice for a bunch of desktop Windows OS systems but you might as well just assign them each a public IP. If it were me, I'd start looking for a different chat service or set up my own XMPP server and keep the Windows hosts out of public IP space..

-Gary