# Dante- Can Ping cards but Dante Controller isn't seeing them



## StradivariusBone (Jun 24, 2020)

We're building a dante network between two venues and a broadcast room. We are having trouble seeing the board furthest from the broadcast. It seems like it works intermittently. I can ping all the devices from anywhere on the network. I subscribed some channels from the far board to the broadcast and they still worked, despite dante controller not being able to see it. 

The network topology is very flat. Everything is on the same subnet and there are only 3 switch hops between buildings. The run goes over fiber between the buildings. What would cause the controller to fail from discovering the devices? A lot of the time when it does list all of the devices, information about the device will be unpopulated. 

My thought is leaning towards something in the network filtering certain ports. That would explain why the audio is being passed (if it gets the chance to successfully subscribe) but other info about the devices is blocked? Anyone had this problem?


----------



## StradivariusBone (Jun 24, 2020)

Here's the other fun part. I'm running DVS on a computer in the broadcast room. That DVS computer can be seen by controller at both ends of the campus. So why would DVS work consistently but I'm losing the cards in the mixers from point to point? M32's in both venues and an X32 producer in the broadcast room.


----------



## StradivariusBone (Jun 24, 2020)

I can also see the latency stats for the board (called "Sanctuary") since there are still connections subscribed, I just cannot consistently change them through the controller. This is weird. I should also point out that when I had the computer audio channel subscribed (when I could change it) it worked flawlessly from an audio standpoint.


----------



## sk8rsdad (Jun 24, 2020)

In Dante Controller, I can't see my Dante enabled device. | Audinate | FAQs

In Dante Controller, I can’t see my Dante enabled device.


www.audinate.com





The last 2 bullets are worth investigating.


----------



## MNicolai (Jun 24, 2020)

Usually when I've had issues with segments of Dante networks falling offline it has something to do with a network switch configuration -- usually multicast or QoS related but can also be other configuration-related issues.

Other thing to look out for in the event you have a redundant network is make sure that the primary and secondary networks aren't getting crisscrossed or bridged somewhere along the way. Only takes one device in switched/bridged mode to take down both the primary and secondary of a "redundant" network.

In this case if you're crossing between multiple manufacturers' switches it's also possible that there's an issue with that handoff. It should not be taken for granted that all switch manufacturers will treat traffic the same way and that can create issues at the transitions where each side of the network may intermittently see the other side but not be able to subscribe to reliably.


----------



## StradivariusBone (Jun 24, 2020)

Yeah I have no idea what's between the buildings. Traceroute shows no routers, but three switch hops and latency that's well within Dante specs. That said I'm thinking there are some consumer grade switches involved here. I don't understand why it would still allow the subscribed stream through and not "see" the device enough to modify subscriptions consistently.

This is all primary network for now. Our infrastructure doesn't allow for the creation of a redundant network at the moment.


----------



## StradivariusBone (Jun 24, 2020)

So I checked out the layout and it looks like mostly Netgear switches, and it hops from our building to adjacent building on fiber, goes through a switch and then pops to the next building on fiber, and then to the device. I pulled this slide from the Dante level 3 training (which I had time to actually complete cuz corona)

We had some IP address conflicts the day we brought the network online and our IT guy was having trouble with the DHCP. I'm wondering if he might've locked down ports or something in his process to fix that. I'm waiting for an email back from him now.


----------



## MRW Lights (Jun 25, 2020)

Remember from the training that multicast subscriptions can cause issues in switches using snmp snooping. DHCP should be okay, but you're likely looking at an uplink trunk conflict in your IP addressing scheme between switches/buildings. Without using DDM crossing subnets is impossible. You may be using the same subnet locally, but when you leave your network your uplink may be blocking your ports from communicating across your backbone. 

Your setup should work fine. We have a very similar setup, 4 buildings, several venues and master control. We keep our Dante network isolated to our own switches that don't cross with anything else. Can you move your network to your own dedicated switches?


----------



## StradivariusBone (Jun 25, 2020)

So far as I have been told, the whole campus is on the same subnet. When I asked if we could get our gear vlan-ed together I was met with a little resistance. After going back and forth a bit he didn't really understand the type of loading we are planning on putting on the network (also running NDI for video). 

I'm leaning towards something is dumping the multicast traffic since when I was able to subscribe audio it worked beautifully, but only in unicast. I can't see past the main building to the other venue's board in controller, but all the data stuff about the devices is handled through the multicast ports. The IGMP snooping might be part of that. I can ping all the Dante hosts from anywhere on the campus, but anything related to multicast is wonky. I'm about to head over again, had to put my carp hat on this morning to build a camera platform. 

Felt good to smell cut 3/4" ply again...


----------



## StradivariusBone (Jun 25, 2020)

It's working better today. I still can't see the far side of the world, but earlier I was able to see it and subscribed to a channel just playing music. That's been running consistently, but I had clock sync issues with the far one. I set it to preferred master and it seems to have settled down, but now I can't see it and I keep getting notifications that a device named "001DC11A4AD6" is elevating to grand master, which I can only assume is the lost sheep Dante M32. 

I'm getting audio from it, it's responding to pings from all the 224.0 addresses (but it wasn't for a while). I feel like there is something still on the network that is messing with the multicast traffic.


----------



## macsound (Jun 30, 2020)

Are you allowed to use wirecast on your network? Sometimes school and corporate people discourage it.
Put a laptop at the switch that relays the data between buildings, check the time on the laptop to see it's accurate, then unplug and replug the dante on the M32 while remote desktoped into your controller that's in the other building. Make note of the time as well as if you can see the device, if it disappears and reappears or anything else. 
Then go back to your wirecast machine and take a look at the logs based on the time you did you plugging, see if there's any egregious data like a reset.


----------



## TimMc (Jul 1, 2020)

You mean Wireshark?


----------



## macsound (Jul 1, 2020)

TimMc said:


> You mean Wireshark?


Oh goodness. Wireshark, yes, thank you.


----------



## StradivariusBone (Jul 1, 2020)

Yeah, I think we can try that. So a bit of an update. We are using NDI for the video feed (1 out of a media computer and 2 cameras) from the remote venue to the broadcast, but we were unable to get the Dante stable enough for broadcast by last Sunday. What we did as a workaround was feed an out to one of the cameras with balanced inputs and piggybacked the audio along the NDI to get it to work. Wasn't ideal, but we also didn't have to cancel. Meanwhile, the NDI stuff (all Magewell gear) worked flawlessly and barely taxed network traffic.

Here's what I found out, because I realized I haven't updated this thread recently. The ability of Dante controller to see and interact with devices was intermittent, but it was consistent in that we could always see the close venue (we'll call it A) from the broadcast room and the far (B) was hit or miss. But when using Dante controller in the B venue booth we could see the B dante device, but not any of the A or broadcast room gak. The absolute wildcard was my laptop running Dante Virtual Soundcard. That would show up wherever, but anytime I subscribed audio it would either give me a triangle of death or it would initiate a cycle of clock warnings and failures. 

So, we can ping all devices from anywhere, but what we can't do is ping mDNS and get replies from all of the devices from everywhere. And mDNS is how Dante handles discovery. So while the unicast audio would occasionally work, the random times I was able to subscribe anything from B, the clock would be unstable. 

Fast forward to my conversation with our tech today. To be fair, he inherited this little network and it hasn't been more than 20ish computers between three buildings, a server and some printers until we showed up and burned everything down with COVID-induced technology. He eventually wants to assign static IP's to all of our devices and figure out some infrastructure plotting to clean up things. But he wasn't quite sure of any policy in place that would explain the dropped mDNS requests. So I asked him if I could look at the switches configs and he was cool with it. My boss already poked around a bit and discovered ICMP snooping enabled on one of the switches, so I'm hoping this is an easy fix now of just figuring out what policy is in place that's screwing it up. It now seems like the classic IGMP dumping the multicast into the wrong ports or just blocking it from the fiber uplink or something.

Also, Wireshark is awesome. That's saved my bacon a couple of times in the past.


----------



## macsound (Jul 1, 2020)

Not sure if there's really only the 3 switches you mention, but something super handy about Unifi gear is you can see the topology live when using the controller. Seems based on your setup, you're not too strapped for cash, take a look at the unifi demo in a chrome browser. https://demo.ui.com
It's a little tough to see because they don't have any fake devices, but essentially, you can find all the hosts and the route they're taking, visually.


----------



## StradivariusBone (Jul 2, 2020)

Unfortunately we have the Unifi switch in the broadcast room and everything else is a mix of Netgear or similar-grade hardware. The switches in the IDF's and MDF are all managed, but yeah. I did poke around on the Unifi controller and it does show all the clients and whatnot and has a really cool interface. I will definitely recommend that when we do upgrade we add more of them.


----------



## StradivariusBone (Jul 3, 2020)

A couple of updates- 

Accessing all the switches I was able to see I found a few running IGMP Snooping, I first tried enabling all of them and no effect. Then disabling all and we were able to see the other building! However, the clock was still unstable. Watching the far building device on controller it would swing high and low and lose sync periodically, maybe every couple of minutes. But we were able to get device info and subscribe audio. 

At the IDF between the far building and the MDF there are a couple of media converters that are only 100Mbit. I'm not exactly sure how they have the fiber run (because there's actually two pairs running between each building), but would that be a potential bottleneck that's messing with the clock? Going to try some more work on Sunday, but was wondering if that was a red herring or not.


----------



## MNicolai (Jul 3, 2020)

100Mbit media converters are likely to create an issue. Off the top of my head, I think Dante only supports 48 channels over a 100Mbit connection, and if those links serve other purposes for the campus network then that will eat into that even more.


----------



## StradivariusBone (Jul 3, 2020)

Yeah we're only dumping at most 32, but try to keep that under. No sense subscribing to a channel that's unplugged. The network is a shared one though and in addition to the Dante, it's also running IP security cameras, access points, and the NDI video feeds. If a bottleneck would affect clock that might be another issue. The stream itself when working was fine. The clock just kept dropping out.


----------



## MNicolai (Jul 3, 2020)

IP cameras, WiFi, and video feeds will certainly bog down that 100Mbit link pretty quickly. I wouldn't be surprised if those are causing issues. It may not be the entire issue and you may need to resolve some switch configuration settings, but I would start there because that should be pretty inexpensive to upgrade.

You might be able to log into one of the switches and see how much bandwidth is moving across the link -- some will tell you, and others won't, but if the link the is saturated then your traffic is getting crunched by the QoS.

If you want to troubleshoot around it, I would look closer at all of the switch settings along the signal chain -- verify multicast settings, IGMP snooping on, QoS enabled with DSCP priority levels assigned, IEEE energy efficiency off, etc. This Yamaha page has examples of how that would be configured for Cisco SG300 switches. Some of the terminology would be different for someone else's switches but the concepts are generally the same. Here's Biamp's version of that for Netgear. As a shared network, you want to be careful with QoS though -- don't assume your network coordinator will be happy with throwing everyone to priority level 0 like the Yamaha guide would suggest. The best case scenario is to not saturate the bandwidth of the link in which case QoS doesn't come into play.


----------



## StradivariusBone (Jul 4, 2020)

That Netgear guide is great. Everything past the Unifi switch in the broadcast room is Netgear, a mix of 24 port and 8/10 port switches. All of them have IGMP support and QoS, but not all of them have very many settings that go that deep. I think only the 24 port guys let you even modify the queues for the DSCP tags, and even then I didn't see a way to add tags for it to recognize. 

There's 6 strand fiber between the buildings and 2 pair are being used. The one fiber link is on a netgear switch with fiber connectors and it is a gigabit connection. The other one isn't (media converters, etc), but I'm not sure how those interact, it looks like there might be two 24 port switches in the MDF and the two fiber runs are both being used by each? I'm guessing both are being used to make a bigger pipe overall between the buildings, but again it's all on the same subnet. 

It's been about 20 years since I've been in a networking infrastructure class, so I'm a bit rusty on what flies these days. Since there's an open pair of fiber on these runs I'm wondering if we couldn't just isolate all of our AV traffic to that pair. Can you patch fiber together at the rack? (I vaguely recall learning about how to splice it and taking a class that explained how bad it is to get fiber optic into your bloodstream, again like 20 years ago.) Or would you need some kinda SFP switch to connect those? The runs are not terribly long. This is our backup plan assuming configuration fails.


----------



## FMEng (Jul 4, 2020)

Yes, fiber strands can be patched together. It's just a matter of matching connectors and strand types. There is single mode and multimode, and you cannot mix the two. Single mode usually has yellow patch cords and multi is orange. Often, the patch panel has one kind of connector, and the switch has another, but patch cords come in varieties.


----------



## StradivariusBone (Jul 5, 2020)

Pretty sure it's all multimode. Had some luck today, but determined that running IGMP Snooping on the switch in the broadcast room will in fact block all multicast data from the uplink port. I had wireshark going watching the PTP packets going and the second it was enabled everything stopped. It was hit or miss on the Netgear switches, which are a hodge-podge of make and model, but none of them have that many settings to modify. In fact, it turns out that Ubiquiti switch can't even do QoS and the VLAN capability is pretty limited as well. 

Was able to see everything and make subscription changes with devices, but still experiencing PTP clock dropout about every 2-5 minutes from the long run with only a couple channels subscribed. I think it's odd that IGMP shut down the traffic, but I don't know how it actually works on that Ubiquiti switch. 

The plan right now is to take over that last unused pair of fiber, patch together in the middle, stick media converters on either side and a hardware router in the MDF to connect, but segregate it from the rest of the network. I can only assume that the clock drop out is from some other conflicting traffic on the network, but I can't seem to figure out what it is.


----------



## MNicolai (Jul 5, 2020)

See this article on IGMP. In short, you want it enabled otherwise your multicast traffic gets sent to many other segments of your network. The change you're seeing where it "blocks" the traffic probably isn't the case. What's happening is that the endpoint devices aren't making a successful subscription or the multicast pool isn't being advertised across the network for those devices to see there is something to subscribe to. Disabling IGMP to get traffic flowing is really just a band aid around a larger multicast issue and without it enabled, your Dante traffic will bog down other areas of the network because it opens the rest of the network up to those packets whether the other devices on the network are asking for them or not.


https://service.shure.com/Service/s/article/dante-networks-and-igmp-snooping?language=en_US



Moving to an isolated AV network is probably the way to go. Underlying problem could be a switch config issue but if the network is all different types of switches then the switch configs may not behave as intended even if they are set correctly.


----------



## StradivariusBone (Jul 5, 2020)

I watched mDNS and PTP traffic packets completely stop interacting with the switch with it enabled. I thought it maybe was a discovery period where it had to setup the internal tables of which ports had hosts that would reply to multicast, but it was a hard switch. Dante controller immediately lost sight of all devices, including the board sitting next to the switch. I've read that Shure article, but from what I've seen with Audinate's guides IGMP only really becomes a dealbreaker when you're dealing with a much bigger network and crossing subnets. Now, I have no clue what the traffic from the security cameras is. I assume unicast RTSP, but I don't know. If they are like the ones I run at my house, they stream constantly and use the server to "trigger" for recording and consume more bandwidth on the network, but I think it's a proprietary setup so I'm thinking the cameras only initiate the stream when they detect? No clue though. IGMP is definitely ideal for the overall health of the network though. 

Apparently there are some issues with how Unifi switches handle this and other multicast traffic- https://community.ui.com/questions/...n/f297dd0e-c424-4867-9f40-f4234a79e39c?page=1

There was an interesting comment in that thread where the Unifi has trouble going from 100 to 1G, our one venue has a 10/100 switch that we've been looking at switching out to a gigabit. There's just so many variables here, but how that Unifi is handling traffic through it's uplink to the rest of the network seems to be part of it.


----------



## MNicolai (Jul 5, 2020)

I don't know how much further I would troubleshoot. It sounds like you need to upgrade the network to all Gigabit either isolated from the campus or get the campus to upgrade their switches. Should keep everything in the same model lineup, make sure all the switches are on the same firmware rev, and all switches running the same config file, etc. Everything should be Layer 2 Managed at a minimum.

If you are going to continue troubleshooting, I would try breaking the network into smaller pieces, stabilize one end of the network disconnected from the larger network and keep moving across the network to see at what point the Dante traffic breaks (may be hard to do since this probably requires messing with the campus LAN). Or, swap out the Ubiquiti with a Netgear and see if it starts to play any nicer. Gigabit switches aren't that expensive and your IT dept probably has a spare switch sitting on a shelf or in a rack that you can at least test with.

The perfect scenario is having AV on an isolated network. The next best scenario is having AV on its own infrastructure that is tied into the campus network -- but the campus network isn't inserted in the critical signal path for streaming media traffic. The next best scenario is piggybacking off a campus network that's on a uniform set of switches and configs. I generally would not recommend being on the network you've described where it's a hodgepodge of different makes/models/configs/bandwidths.

If you are purchasing new switches, the Cisco SG350 series is the industry standard for Dante. Certainly many other makes/models of switches will work, but the SG350 series poses the least amount of risk and with the SFP slots you can directly tie into that fiber run.


----------



## Rob (Jul 6, 2020)

Have you considered that the switch doing your IGMP queries may not support the number of devices snooping? My colleague Wayne Howell (Artistic License) wrote three good articles in Lighting & Sound International magazine on this subject and some test tools available for download here. 

For those watching this thread but need a bit more info on IGMP, both sACN or Dante make good use of it. I did two videos explaining the basics. The first is a primer where the second is a real world example. 

If you're interested in why an Entertainment Class Switch is sometimes a good choice when building networks, vs choosing Enterprise Class switches, watch my video called Why VIA?. I discuss, among other things, why and what Wireshark does and how it helps to debug some of these issues.


----------



## StradivariusBone (Jul 6, 2020)

That's a possibility. All I know is that with IGMP on with the Unifi switch, zero multicast traffic was allowed through the uplink port. I could not find a way to mitigate that through settings. I believe the rest of the switches on the network still have it enabled at this point. Only two of them have quieriers. 

The plan now I think is to put media converters on either end of the fiber run. Go from broadcast room switch--->media converter--->fiber--->patch at the midpoint IDF--->fiber--->Media Converter--->booth switch. And then get an edge router in the broadcast room to connect to the main network. That way we cut out some switch hops and distance ourselves from the other traffic on the network.


----------



## rphilip (Jul 6, 2020)

StradivariusBone said:


> That's a possibility. All I know is that with IGMP on with the Unifi switch, zero multicast traffic was allowed through the uplink port. I could not find a way to mitigate that through settings. I believe the rest of the switches on the network still have it enabled at this point. Only two of them have quieriers.
> 
> The plan now I think is to put media converters on either end of the fiber run. Go from broadcast room switch--->media converter--->fiber--->patch at the midpoint IDF--->fiber--->Media Converter--->booth switch. And then get an edge router in the broadcast room to connect to the main network. That way we cut out some switch hops and distance ourselves from the other traffic on the network.


Double check but I think you only want one IGMP quierier on.

Your plan of getting semi-isolated make a lot of sense and will probably clear up your problems and is a better solution.

Philip


----------



## StradivariusBone (Jul 6, 2020)

The quierier apparently uses an election process not that dissimilar to the dante clock elections. I set one as the primary and the other as a secondary, but ideally the switches will decide which one is the better. Again, these are regular Netgear switches without many bells and whistles so it's anyones guess how well they handle it.


----------



## Rob (Jul 6, 2020)

StradivariusBone said:


> The quierier apparently uses an election process not that dissimilar to the dante clock elections.



That is how our VIA switches work. We state in the manual that you should have _at least_ two quieriers and snooping turned on for every VLAN that may have multicast devices attached.


----------



## Ben Stiegler (Jul 8, 2020)

Try turning off all the non Dante equipment as an experiment. Kill the cameras, etc. if the Dante clock is happier, you have your answer re 100Mb bottleneck. QoS adjustments might also save you ... losing a few frames from a parking lot cam might be acceptable til you can get faster media converters. Also, let's not assume that the fiber was terminated cleanly. Many moons ago I did a training with Sprint North Supply (anyone remember them? Early distro giant)band it was not easy to do it right. Now imagine some electrician who watched a video the night before doing his first fiber terms on your campus...

you might also try swapping fiber pairs if there are spares.

keep us posted!


----------



## Ben Stiegler (Jul 8, 2020)

StradivariusBone said:


> The quierier apparently uses an election process not that dissimilar to the dante clock elections. I set one as the primary and the other as a secondary, but ideally the switches will decide which one is the better. Again, these are regular Netgear switches without many bells and whistles so it's anyones guess how well they handle it.


Have you checked that the switch firmware is up to date and consistent across all the Netgears?


----------



## FMEng (Jul 8, 2020)

It doesn't seem to me that bad fiber termination would reduce the bandwidth. It's either above the minimum light level and working, or it's not working at all.


----------



## StradivariusBone (Jul 8, 2020)

The firmware is a great point. I am almost certain they are not all up to date. And I'm afraid to attempt shutting down the network to do isolation troubleshooting. There's not much in the way of documentation on the infrastructure. I'm also afraid to update the switch firmware for similar reasons lol. 

I'm hoping we can just use the two unused strands on the trunk lines and isolate it. We're going to try that this weekend. If not? Then we might have to go that route


----------



## StradivariusBone (Jul 21, 2020)

A couple updates. We replaced one fast ethernet switch with a gigabit switch. This is the one that networked an Aviom/Dante box with an M32 in the venue we were able to link up to. It vastly improved life, the universe and everything between the broadcast room and the near venue. It also seemed to help out on some level at keeping the clock happier between the far building and broadcast, but we are still experiencing the occasional clock dropouts.

Fiber links between buildings is all OM1, and we got some media converters to try and utilize the unused pair, but with no success. I don't have any test equipment so the best I could do was polish the ends and try to get a link with the media converters on either segment, neither of which was successful. 

It's a 6 strand fiber run, 4 of 6 are used. One link is gigabit, but the other is 10/100. After seeing the improvements from swapping the local switch with a gigabit I'm willing to bet that the traffic might be getting dumped over the slower link, but it's hard to tell with how the switches are configured on both ends. 

Looking into a couple of options today. Trying to see if the distances between IDF's is under 300' and if we could just get away with running copper. If not, it's either upgrading the 10/100 media converters to something better, running new fiber, or crying in the corner. We're also going to attempt to repair that open pair on the existing fiber. 

Thanks for all the help so far, gang. I just wanted to update to hopefully lead to some closure on this project.


----------



## FMEng (Jul 21, 2020)

Did you make the required crossover for the media converters? Tx to Rx. I've seen fiber installers do some funny things when they terminate cables.


----------



## StradivariusBone (Jul 21, 2020)

Yeah, I tried both ways just to be sure. Here's the fun part. So MDF to IDF 1 has two fiber links (four strands out of six total being used). The one link is 10/100, the other was going out an SFP on a gigabit switch. In IDF that first link terminated into another 10/100 transceiver and then into a switch. The second link went into another switch that looks optimized for the access points we use. There are additional links between IDF 1 to IDF 2 (which is where the venue we're trying to hit is). The first link was 10/100 and terminated in IDF 2 into an ancient 10/100 switch with only one active host connected to it. Turns out that it was the AC controller, so we'll leave that be. The second fiber link between IDF 1 and 2 was a gigabit connection and terminated into the switch we were using to connect to the venue Dante card. So theoretically, we had a good fast link between IDF 1 and 2, but on IDF 1 to the MDF we were only going over 10/100. 

I took our gigabit media converters and replaced the 10/100 ones on that link b/w MDF and IDF 1 so now the pipe should be gigabit all the way through. We were able to get link and talk on the devices happily for about 10 minutes and then as we turned on devices we started to have clock issues again. I took a step back and went conduit hunting, eventually we want to isolate the whole network which looks like it will mean new fiber. I came back to the broadcast room and found that the clock sync was dropping periodically. 

I went back to the switch configs and noticed that we still had all of them running without IGMP since previously it seemed more stable. I went ahead and turned that all back on (except for the Unifi switch in the broadcast room- there's some issue with how IGMP snooping works on that hardware, it kills all multicast traffic). And so far....so good.

I'm hoping that we were still flooding the switches with clock sync packets and maybe that was causing it to lose clock? Now that we're gigabit all the way down hopefully that helps out. 

We are still seeming to have trouble with an Aviom D400 with the dante card. Turning that thing on seemed to aggravate the clock situation, but we'll get to that in a bit. Right now I'm just letting it sync between the venue at IDF 2 and our broadcast room and it's pretty happy, at least it has been since I started typing. Just checked, it's still happy. I feel like a Bob Ross network analyst. Happy little link lights. 

Fingers crossed.


----------



## StradivariusBone (Jul 21, 2020)

Well the clock has been stable for about 30 minutes but I can no longer see the Dante card out past IDF 2. Still getting audio though.


----------



## StradivariusBone (Aug 6, 2020)

Well! Have I got some news! On a separate note we managed to get some PTZ cameras in after them being backordered for months. I was helping my boss get those up and running when it just struck me once again that it was ridiculous that we could stream 3 HD video feeds down this pipe with minimal latency, send PTZ control info back up the pipe with minimal latency, throw VOIP Mumble chat everywhere on this network...

...and this dadgum Dante was still just not cooperating. 

So I bit the bullet and went into the switch firmware. Initially I was hesitant to do this in a live production network (we share this with the normal staff of the church) but I started updating them one at a time, going from the furthest to the closest. I got as far as the IDF and all of a sudden everything started clicking. It turns out that one of the switches at the end of a fiber run had a bug. Here's the snippet from the release notes-

"When the device receives hundreds of Pulse Audio multicasts (with destination IP of 224.0.0.56) per second, GUI access and Remote Diagnostic are blocked."

Now I can't find anything that says that Dante specifically uses that multicast port, however Dante does dump A LOT of multicast traffic and this issue explains the issues we would experience where it would initially discover the board, eventually lose the device info and then ultimately lose all ability to subscribe audio, in spite of still happily transmitting the audio streams already subscribed in between fits of clock failure. I'm guessing whenever we turned the board on, it would begin to overwhelm the bugged switch with multicast packets until it finally shut down and blocked all the 224.0.0.0 addresses coming and going, but it didn't affect any of the unicast or broadcast messages, so all the rest of the normal traffic just kept on trucking along.

I need a beer.


----------



## FMEng (Aug 6, 2020)

Wow, fantastic job persevering and troubleshooting. Many IT problems can only be solved by being stubborn enough to keep doing trial and error. IP audio and video is very cool when it works like it should.


----------



## StradivariusBone (Aug 7, 2020)

Thanks! And thanks as always to you all for adding your $0.02 as I worked through this. @Ben Stiegler wins the prize for the final piece of the puzzle here. Hopefully someone in the future will find this useful too!


----------



## StradivariusBone (Aug 22, 2020)

So. Hopefully FINAL update on this one. It turns out the fix we thought was the fix was not in fact fixy enough to be the fix. 

In retrospect, I now believe that the the intermittent issues we had might be related to the addition of the IP security cameras on our campus and a lack of VLAN capability with the existing switches. IP cameras as I understand them will vary the bandwidth used from idle states to triggering states. I know from my home setup using Zoneminder, I can configure them to output the full stream or setup a secondary stream with lower framerate and there are ways to modify the camera itself to initiate the alarm state vs. the server. So there's a potential for a lot of bandwidth to be used when the building is occupied vs. when there's no one around. Hence, I think that might be why it worked great in the evenings when most of the campus was quiet, vs Sunday mornings when there were more bustling people.

Anyways, we punted. We invested in 300' of conduit and 550' of fiber. We pulled the fiber in Thursday and earlier today we patched everything in and I can honestly say I've never been happier to see a little green light come up. So now our setup includes a broadcast room switch that connects to the nearby booth via copper to a booth switch and the far booth via fiber to another switch. We put a hardware router in between our main switch and the network, so now I have control over DHCP and whatever traffic passes over our network to the larger network. 

Of course. Tomorrow is Sunday. I might be posting this celebratory post prematurely, but when I left earlier this afternoon we had cameras and audio from all corners of our little world. Fingers crossed.


----------



## StradivariusBone (Aug 24, 2020)

Sunday came and went. 100% functional. It's Miller time.


----------

