# Network Administrator introduces a new wrinkle



## teqniqal (Feb 28, 2019)

This post affects anyone using networks, so if the moderator can cross-post this to the Sound and Lighting groups as well, it would be appreciated.

I just came from a meeting for a school project where the School District IT department wanted to have a coordination meeting with the sound and lighting contractors regarding the Wireless Access Points (WAPs) for the two systems.

Background:

We specified that the dedicated and _fully isolated_ network for the digital audio and A/V controls have a dedicated WAP to allow the owner's sound staff (and authorized students) be able to remotely control the sound mixing console and Dante network with a laptop, tablet, or mobile phone with an app.
We specified that the dedicated and _fully isolated_ network for the dimming system and controls have a dedicated WAP to allow the owner's lighting staff (and authorized students) be able to remotely control the lighting console from their tablet or mobile phone with an app (remote focus function).
The IT department is deploying a new system to all of the schools, and in the school wide WAP system (that is also in the auditorium), the new 802.11 ax WAPs are able to self-regulate their power levels and are programmed to 'seek and destroy' (overpower) any and all non-authorized WiFi devices (yes, a bit Orwellian for my taste). It would see my two dedicated WAPs as enemies and overpower them. So, they want to abandon our WAPs and connect their WAPs to our two isolated dedicated network systems through a heavily managed Cisco switch. They say they can isolate our two networks as we have requested, and only let the dedicated devices connect through to our two respective systems. I get it -- its all about security and threat reduction.

The difficulties I see are this:

If their system goes down, our local remote control of our systems are screwed.
Bluetooth also operates in the WiFi band, so will this 'seek and destroy' mess-up simple things like Bluetooth wireless keyboards, mice, and headsets? Will it disrupt a wireless sound feed from a guest's phone into our sound system?

Any WiFi band (2.4 or 5GHz) type DMX extenders become useless in this environment.
Any casual use of a laptop or phone to create a local temporary WiFi hotspot is shot to hell.
WiFi links between camera memory cards and bulk storage or a laptop will be shot to hell.

Any guest artist that might bring a WiFi or Bluetooth connection between their own equipment will be hammered into non-usability.

We are wanting to keep the A/V and lighting networks TOTALLY isolated so software can't and won't automatically update, but it is my understanding that MS Windows (10) has some curious 'sub channel' tunneling is does to get updates and it is difficult to kill this activity (_persistent little MF!_). Truth? or rumor?
So, I am curious, has anyone else encountered this?

What other _gotcha_'s might I need to look-out for?

Open for comment. The systems aren't up and flying yet, but will be within a month or so. Fingers crossed.


----------



## dmx (Feb 28, 2019)

Good Morning!

Is there any chance that perhaps the board member introducing the system was being a little overzealous? WiFi blocking is an illegal practice as defined by the FCC:
https://www.fcc.gov/document/warning-wi-fi-blocking-prohibited

Heavy fines have been levied against alleged offenders:
https://docs.fcc.gov/public/attachments/FCC-15-146A1.pdf

Hope this helps your cause!
-Matt


----------



## Scarrgo (Feb 28, 2019)

We had this problem to a small degree, but working with our IT, showing them that our system does not want to see any sort of filtering or any of that voodoo that they do on our lines.

Showing them that they could not get it to work running thru their system, we had so many drops that it became unusable within 5 min, and we were the only ones in the room.

I also tried to get them to not offer free wifi in the theater, makes it even easier for them, they didn't like that option....

In the end, they said go ahead and run your own wifi system(we run two, lights/sound) and they just ignore us...they also did not turn on the hunter/killing option...

It is sometimes tough dealing with some IT folks, they look at the world thru blinders sometimes, and think that because they have certificates, they think they know more than anyone else.

My feelings about IT, they should be working with/for us, not as there own little kingdom...

Good luck...

Sean...

*edit: spelling/edit line "with/for us" as sometimes we are also not as smart as "we" think we are(speaking of myself)


----------



## Calc (Feb 28, 2019)

We ended up with a similar system- here's what we have:
Independent network switch for show network devices. All cable runs lead back to our switch in the booth, not the regular network closets. We aren't running Dante or pushing content over the network, so we left sound/lighting/video all on the same network.
The switch in the booth is then connected to the campus network. The university has that port relegated to it's own VLAN that is only accessible to certain users, so random strangers can't get in. It's inaccessible from the internet, obviously.
This lets us connect in from anywhere on the campus network, but only us. If the campus network crashes, our network is still on it's own independent switch so everything keeps running. We'd lose remote access, but nothing else. 

I'm guessing your network guys are being overzealous with their "Seek and Destroy" description. It makes more sense for an access point to AVOID congested frequencies- that's what ours do here. Best-practice for AP's would be to transmit at the minimum necessary power to avoid interference with other nearby friendly AP's. It's possible that they're set to boost their transmit power if they see something else there, I doubt they're going to start jamming any rogue frequencies. Jamming is illegal in the WiFi bands under FCC Part 15 rules- it's that "devices may not cause interference and must accept interference from other sources" sticker you see on tons of stuff. Unlicensed devices are allowed to overpower each other, but not to actively jam.


----------



## StradivariusBone (Feb 28, 2019)

Any vendors trying to run a hotspot for merch tables would have a problem. My other thought, being a HS TD myself, is what is the personal cell number of the IT contact that I need to call afterhours when this system fails during an event and I've got a client screaming at me because sound and light stopped working?

I've never heard of an AP that blocks other signals. As has been stated I think that goes against a lot of FCC regs.


----------



## MNicolai (Feb 28, 2019)

The Marriott decision is not a universal precedent. They were sending deauth packets to outside SSID's for the purpose of blocking them and generating a profit. The FCC has not made clear a comprehensive list of legal and illegal applications of rogue AP containment and the use of deauth packets, but reading between existing precedents here is my understanding of the landscape:

Probably legal:

Issuing deauth packets to AP's that are trying to mimic your SSID's (direct security threat for man-in-the-middle attack)
Issuing deauth packets to rogue AP's that are on your own LAN (direct security threat for allowing unsecured access to your internal network)
Definitely legal:

Detecting wired switch ports that have rogue AP's connected to them on your LAN, and disabling the switch port thereby severing the rogue AP from your network.
Probably illegal:

Issuing deauth packets to AP's which are not on your own LAN and do not pose a security risk to your network, such as personal hotspots.
Definitely illegal:

Signal interference against other radio-based FCC approved devices (Bluetooth, Zigbee, non-WiFi devices that also operate in 2.4/5GHz, etc.)
Also, this is not how Cisco's "Air Marshal" feature works. It works by transmitting a WiFi packet that other WiFi devices must receive and interpret. Non-WiFi devices would not be impacted by this particular security feature.


----------



## Jay Ashworth (Feb 28, 2019)

My professional opinion, coming from both production and IT background, is that you MUST have the highest administrative official above the Production facility/staff explain to the highest administrative official above the IT staff that their proposed solution will not allow you to provide acceptable service to your customers, whether they're paying in cash or district funny-money, and that they're going to have to find a way to make an exception to that policy.

It's not a technical decision. It's a business decision driven by technical requirements.

Expect to possibly have to explain to your colonel why, before he goes off to do battle with his counterpart. 

That can be expanded a lot, but I'm coming off a 12 hour shift in an 80 hour week, and I don't have the energy.


----------



## macsound (Mar 1, 2019)

Having worked at many schools that have a policy for everything, usually there isn't anyone high enough up to make a decision that overrides a districtwide capital improvement, which is what I assume this is.
When you get honest with the local IT guy, he will probably tell you to do what you need to in order to make what you need work, as long as he doesn't get in trouble and it isn't obvious if ever there was an inspection.

You can easily test the search and destroy protocol they have, but my understanding is they need to be on the same wired network in order to work. They can't kill rogue access points that are just closeby. 

So you should easily be able to run your own wired network and have your own physically hidden APs. Like put them in a black cardboard box in the catwalk. 
I'd recommend making the ssid hidden and obscure like Netgear123, just in case there is an inspection.


----------



## Jay Ashworth (Mar 1, 2019)

No, Mac; I've worked with what they're talking about.

Building Wireless Management stuff knows which SSID's it's providing, and it listens over-air for connections to other SSIDs, and stomps them out by deassociating.

To avoid that, someone with admin access to the wifi system config is going to have to add the production SSIDs to the whitelist; there's no real way to skate it.


----------



## macsound (Mar 1, 2019)

I'd love to know what company offers a product like that. I know Meraki and Ubiquiti has to disable the active network in order to be in defensive mode.

Also, thinking more into getting around the IT team, might be worth testing using 802.11a. Most modern access points don't support it anymore and therefore might not be able to squash it.


----------



## Jay Ashworth (Mar 1, 2019)

Really? I just put in some UBNT dual bands last year, they still supported 11a.


----------



## Lyle Williams (Mar 2, 2019)

Transmitting for the express purpose of preventing intercommunication (even jamming your own system) is probably illegal.

Sending your own equipment an intelligible packet telling it to stop will be OK. 

And yes, this needs to get escalated. There may need to be agreements like no institutional (ie, non-show) information on the show network. IT are trying to protect personal and corporate information from disclosure via inadequately managed rougue networks.


----------



## Lyle Williams (Mar 2, 2019)

The number of SSIDs on a campus will also be huge, with every second smartphone providing a personal hotspot.

Good luck to anyone who wants to track and manage that.


----------



## Jay Ashworth (Mar 4, 2019)

This isn't a manufacturer reference, but note in this article that the issue we're discussion -- systems to look for rogue APs and attack their connections -- is *recommended* by PCI DSS (the credit card security council) for networks that process card transactions:

https://en.wikipedia.org/wiki/Wireless_intrusion_prevention_system


----------



## Jay Ashworth (Mar 4, 2019)

Manufacturers include Watchguard, Netscout, Cisco, and Aruba... at least.

See also this article, and its definition of "prevention":

https://www.cso.com.au/article/611681/selecting-best-wireless-intrusion-protection-system/

Yes, there are commercial systems that you can tell which SSID's are "Acceptable", and they'll break the other ones.


----------



## jtweigandt (Mar 4, 2019)

You might have a discussion with the IT folks as to what "overpowering" means. Most goods of this nature operating in "unlicensed bands" are required
by the FCC to not generate interference.. and to accept interference, so if it's truely an overpowering of a band or channel, uh that's illegal dude... I can see where the system 
might blacklist a mac address, and that's a more likely scenario.. but I'd hold their feet to the fire and make them define their terms. If it's truly 
an overpowering situation... might remind them that anyone complaining about it to the FCC could cost the district some big fines. There already have
been cases of Theaters and other locales that jammed cell phone signals, and got in hot water for it. Someone may be playing fast and loose with the rules here. 
If we had to dump a bunch of wireless mics at the feet of the FCC, it's only fair that the wifi comply with the rules too.


----------



## Jay Ashworth (Mar 4, 2019)

Nope. Deassociate frames are a *sanctioned* method of dropping clients from wifi networks.

Really: read the article, JT.

Got nothing to do with "jamming", as that term of art is used in wireless communications.


----------



## jtweigandt (Mar 4, 2019)

From the article
"
Wrongful classification of an external AP or client device as rogue and taking action to isolate it can have a number of negative consequences ranging from reputation damage to legal implications.

A good WIPS solution will detect and provide visibility into all APs and client devices on or around an organisation's airspace. By the nature of how Wi-Fi works, even if a client device or AP is not directly connected to an organisation's network, it will still show up as being in its airspace. It is very important, therefore that a business is able to not only see that device but understand if it is truly connected or just within range before they take action against that device or AP.

"
Very few WIPS can accurately classify client devices and APs with low enough false positive or negative rates for admins to have confidence to enable prevention. WIPS that utilise techniques to correlate MAC addresses of client devices seen in the air with MAC addresses seen by network switching equipment are notoriously prone to high false positive rates and rendered useless. The same situation also occurs for WIPS utilising custom IPS detection signatures where manual intervention of tuning and scripting these signatures can result in a unusable WIPS. WIPS that utilise re-broadcast packets both on ethernet cabling and over the air are the most accurate and the ones where automatic prevention can be confidently enabled.

Without accurate classification, the prevention aspect of WIPS will no longer be immediate and instead becomes a manual process for the IT team or department."


So if the IT guys were saying the new stuff would break the standalone existing theater SSID and network... then their detection and reaction ability is not tuned well enough to avoid complaints from someone trying to say run their tablet off their phones hotspot somewhere down the hall while they are selling cookies and punch.


----------



## jtweigandt (Mar 4, 2019)

Now if someone can rig me an EMP rifle to take out squealing audience hearing aids, I’m all in.. FCC be damned


----------



## Jay Ashworth (Mar 4, 2019)

Sure, JT. I absolutely understand where the break between technical capability and administrative control lives.

The *question* was whether such technical capabilities existed at all, and clearly, they do, and our OP might have them imposed on him, whether it violates some law or regulation to do so or not -- he's not imposing the control, so he's off the hook.


----------



## mbrown3039 (Mar 6, 2019)

Thanks for posting this, Erich -- very timely and potentially immensely impactful to all of us. My first suggestion would be something already posted above: no campus WiFi in the theater. 

I have not run into this very scenario, but I recently helped convince a major hospitality firm here in Vegas to create a standard for new construction whereby all new venues (and existing venues, as they are upgraded) must have a dedicated A/V/L network that doesn't touch the corporate network and has a dedicated tunnel to the outside world (via a dedicated cable modem). Educating the corporate IT folks about the protocols show networks employ has also been very helpful: i find that once you mention multicast, port forwarding and IGMP (with snooping!), they quickly agree to a separate, dedicated network.

Good luck, and please keep this thread updated as you sort through this -- I am curious how it all turns out.


----------



## Jay Ashworth (Mar 6, 2019)

Another tag on the end here:

Your network and WLAN configuration will be complicated a bit further: You're going to have production people, who need access to the production-net *and* the Internet, and you'll probably have guests who need Internet access, but shouldn't be allowed to even know the production net exists.

This almost requires a 3-port router with VLAN, and wifi nodes also with VLAN and multi-SSD support, to *really* do it properly.

I would probably do it with a WatchGuard, and UBNT Unifi flying saucers, myself...


----------



## Calc (Mar 6, 2019)

Jay's described situation is what I've ended up with here- My phone ends up on the regular vlan, but my user privileges grant me access to the theatre VLAN. It CAN be done, you just need the network guys to be willing to work with you.


----------



## teqniqal (Mar 6, 2019)

Jay Ashworth said:


> This isn't a manufacturer reference, but note in this article that the issue we're discussion -- systems to look for rogue APs and attack their connections -- is *recommended* by PCI DSS (the credit card security council) for networks that process card transactions:
> 
> https://en.wikipedia.org/wiki/Wireless_intrusion_prevention_system



In the article it says: "WIPS should understand the difference between rogue APs and external (neighbor’s) APs".
This _is_ the issue. The isolated and dedicated Sound and Lighting system WAPS are 'the neighbors', not 'rouge WAPs'. I am beginning to think the System Admin for the School District doesn't get the difference. Well, if their approach doesn't work, I'll fall back to 'splainin' this to them, _again_.

You mentioned Aruba, and yes, this is their system they are using.


macsound said:


> Having worked at many schools that have a policy for everything, usually there isn't anyone high enough up to make a decision that overrides a district wide capital improvement, which is what I assume this is.


 You are correct.


macsound said:


> You can easily test the search and destroy protocol they have, but my understanding is they need to be on the same wired network in order to work. They can't kill rogue access points that are just closeby.


 Yes, this is the situation. The dedicated Lighting and sound WAPs were planned to be 'closeby', not a part of their system.


macsound said:


> you should easily be able to run your own wired network and have your own physically hidden APs. Like put them in a black cardboard box in the catwalk. I'd recommend making the ssid hidden and obscure like Netgear123, just in case there is an inspection.



Well if the cat wasn't already out of the bag that our WAPs would exist, that is what we would recommend. Too late for that, though. They know.

Thanks to everyone for the input. It's helping me clarify the situation. I'll report back once the dust settles.


----------



## Jay Ashworth (Mar 8, 2019)

I don't know that it's that they don't get it, Erich.

I think it's that their policy says "no SSIDs except those specifically authorized"... and no one has authorized them.


----------



## jtweigandt (Mar 9, 2019)

That’s why I was pointing out that THEIR policy of “no SSID’s except those specifically authorized” should not by law and regulation supercede the FCC rules which prohibit active blocking of those SSID’s I would kind of raise that point with them, and the school board, and it might not hurt to remind the school board that active blocking of non intrusive signals has the potential to be expensive for them down the road should someone complain that their hotspot was squelched.


----------



## darinlwebb (Mar 9, 2019)

It's a people problem, not a tech problem. In addition to your technical requirements, you should detail the consequences of not having them. Are we talking inconvenience during rehearsal and programming, or significant risk to the smooth-running of a show? You also need to acknowledge the good reasons backing their plan. An easier-to -manage network, better wireless performance throughout the school, an opportunity to upgrade their aging infrastructure for cheap because this company is offering them a deal.

How can you compromise? Does your wireless network require internet access? If so, assure them people can't use it as a backdoor around their firewalls and filtering. In addition to running your equipment, do the people in your space need/want wifi internet? If so, then you're going to need their hardware to support them because hell if you're going to play helpdesk when some conference attendee can't check Facebook.

You might not be the right person to convince the IT folks leading the charge that the wifi disruption feature is not worth the hassle - but they should be honestly evaluating the actual value here. Are they dazzled by a sales rep into solving a problem they don't even have? If you're stuck with it, team up with your IT crew to badger the vendor into supporting 'zoning' of the seek-and-destroy feature. There are other scenarios besides theatre where you might need ad-hoc wifi networks to pop up.

In any case, make sure they start with your area when they begin the rollout so you can work together to test your requirements ASAP.


----------



## tomthetechie (May 12, 2019)

Has the option of creating SSID's with Audio and Lighting VLAN's without access to the rest of the network, and by extension the internet, been discussed with IT? If they are going ham on security I could see MAC filtering being an obnoxious obstacle but that's far from the end of the world. Are we talking console(s) access only here? Outside of that, I don't personally understand the level of concern raised on this thread. If you have to start integrating AV network traffic onto the venue's network that does bring a different batch of headaches but even then my biggest frustrations with integrated networks comes down to 2 things, low level network engineers who fight with me on required settings (IGMP, QOS, and jumbo packets) and the BS method of discovery Audinate opted for.


----------



## Sparks & Light (May 17, 2019)

teqniqal said:


> Bluetooth also operates in the WiFi band, so will this 'seek and destroy' mess-up simple things like Bluetooth wireless keyboards, mice, and headsets? Will it disrupt a wireless sound feed from a guest's phone into our sound system?



Bluetooth is usually unaffected by the practice you describe. It is specific to WiFi.


> Any WiFi band (2.4 or 5GHz) type DMX extenders become useless in this environment.
> [...]
> 
> WiFi links between camera memory cards and bulk storage or a laptop will be shot to hell.
> Any guest artist that might bring a WiFi or Bluetooth connection between their own equipment will be hammered into non-usability.



True, except for bluetooth.


> Any casual use of a laptop or phone to create a local temporary WiFi hotspot is shot to hell.



That is often one of the goals. Usually, the IT department wants to avoid any sort of WiFi access it cannot control and filter, as a means of making their policy enforcement more effective by controlling gaming, messaging, file sharing, porn, and other prohibited sites.




> What other _gotcha_'s might I need to look-out for?
> 
> Open for comment. The systems aren't up and flying yet, but will be within a month or so. Fingers crossed.



If you are on an isolated segment of their network, then:

Sooner or later they will make a configuration mistake which breaks your system. Perhaps years from now, after personnel changes, when no one remembers why things are the way they are.
Sooner or later they will make a configuration mistake which undoes the isolation of your system.

It would be reasonable for you to ask them to provide an on-call list of IT people who can become involved in troubleshooting if there are problems controlling the lighting. 

Sooner or later there will be an equipment failure that they will not notice because it only affects you.
Any emergency plans (e.g. lighting control during power failure) will have to consider the IT portion of the system.
Your best bet is to emphasize the extra on-call responsibilities, and ongoing planning responsibilities, they will taken on. They can shut off what you call the "seek and destroy" feature on a location by location basis. Have them shut it off in the theater.


----------



## EdSavoie (May 17, 2019)

My two cents is total isolation, and if you need to be attached to their intranet, have a brightly coloured cable (or just cover it with lots of spike tape) "break in case of theatrical meltdown"


----------

