Lessons Learned about Telephone Company T1 Circuits
Do you have problems with your T1 circuits bouncing (going up and down) when there are lightning storms in the area?
See if your local telephone company is using HDSL technology to deliver the circuit on copper to your building. If they are, there's a good chance they're using cards made by Adtran. As this document shows, there is a known problem with early generations of these cards resetting and retraining when there's lightning in the area.
Refer your telephone company engineers to .pdf page 52 of 54 in this Adtran document
Also available for download from this site in case Adtran moves it.
If you get the cards upgraded to a newer version, rev. 3 or newer, you'll see a reduction in the number of short outages due to lightning. Switching to a type of card by PairGain is also known to reduce annoying short outages.
Also make sure that the grounding is done correctly. The cabinet or shelf that holds these cards DOES need to be grounded and bonded to the telecom ground.
The card at your end of the circuit is the HRU (Remote Unit) which can also be referred to as a SmartJack. The one at the CO or RT is the HLU (Local Unit). While troubleshooting a circuit problem, try to get the telephone testers to isolate the problem to either your side or their side of the SmartJack. They'll intrusively test the circuit by looping back your equipment (CPE) and they their equipment. If they test errors while looping your equipment, the the problem could be on either side. If they test no errors looping their equipment but there are errors while looping yours, the problem SEEMS to be yours. BUT, it could be a case of bad timing if this is an intermittent problem. They can't test between just the HRU and CPE.
Once they document that there are errors on their side of the HRU, you're in the clear and they have to clear the problem.
Another tool they can use is PM data. Ask them to 'pull PM data' before they do any testing. That documents that there was a problem, but doesn't do much to indicate where it is. This data shows the number of errors over various time spans. Once they do any testing, this shows as errors, so it contaminates the PM data.
We often have intermittent problems that are hard to track down. We often get 'everything is fine now' when the tech is on-site. Refer them back to the first test, often the autotest, that showed that there is a problem on their side of the demarc. Circuits don't 'heal' themselves, so there's likely still a problem, it's just not acting up while the tech is there.
The type of protector is also important. There was a blue type of protector that was used for a while. It was found open when there really wasn't a problem. The current type of protector is red. The regular black protectors should never be used on one of these HDSL circuits.
Depending on distance, an HDSL circuit can use one or two pairs. The can also be up to 2 repeaters in the span. The Adtran documents above show quite a bit about how HDSL circuits work and can be configured.
HDSL technology powers the card at your building from the card at the CO (Central Office) or RT (Remote Terminal). This puts almost 200 Volts DC on the telephone pair. This high voltage tends to cause pairs to fail if there's even a hint of dampness in the cable. If the telephone techs keep changing cable pairs to fix your circuit, and they fail again a few days later, there's a very good chance you have a cable with water in it. The cable needs to be repaired or the outages will continue.
Generally, this is how I request repairs:
Check power and equipment at both ends. If good:
Open ticket, request "Autotest".
Circuit bouncing, going up and down.
Verify stable power at both ends.
Open ticket. Request 'Pull PM data' before testing.
Wait for call back from tester.