Brent MooreBrent Moore, Director of Customer Support

Last quarter, I discussed various common methods of building redundancy into future Information Technology (IT) plans that would benefit your financial institution (FI).  To reiterate from my previous newsletter, the foundation and primary focus of this article is “planning for failure while minimizing operational impact.”

To summarize my previous topics:

  1. Communications- consider dual Internet connections
    1. Internet outages are common occurrences
    2. Cloud services place larger emphasis on Internet connectivity
    3. Consider each functional area of your FI and ask “What aspects of day-to-day operations would be affected with an Internet outage?”
    4. Duplicate firewall hardware
      1. Active firewall- actively responsible for forwarding traffic
      2. Passive firewall- would assume all responsibility for forwarding traffic in the event of failure
    5. Benefits
      1. Failover- automatic failover to duplicate Internet connection
      2. Failback- once connectivity is restored, the firewall will revert to original configuration
      3. Load balancing- split Internet traffic between two Internet connections
  2. Infrastructure & data redundancy
    1. Virtualization- consider converting physical hardware into virtual
      1. Dramatically enhances recovery options
      2. Gives you the ability to back up the server in its entirety
      3. Recover entire server while troubleshooting
    2. Data redundancy
      1. Local storage- provides a local duplicate copy of your protected data
      2. This gives you the ability to restore from your Local Area Network (LAN) which dramatically improves your restore time objectives vs. restoring via Internet

To continue, let’s consider other areas in your IT infrastructure that could negatively affect your FI if a failure were to occur.  Although computer hardware has made leaps and bounds regarding hardware failure, this is another common occurrence.  I recommend evaluating each workstation and consider its criticality to banking operations.  As a best practice, I suggest keeping already built spare machines on-site as a way to mitigate this risk.  Redundant machines with all the same software/hardware provide almost instant failover if a computer crashes.

Next, take a look at other networking components that could result in prolonged downtime without proper redundancy.  Network switches are essentially the backbone to your LAN.  At minimum, I suggest having half the capacity of your production switches available in case of switch failure.  For example, if you have two 24 port switches for your production network, I recommend having a spare 24 port switch in the event of an outage.  Optimally, it would be best to have enough duplicate switching hardware to operate your production environment entirely.  However, it is worth considering the accessibility of having a switching reseller and/or the ability to mail order the equipment in a timely manner.  Typically, a switch could be overnighted and would be onsite within the next business day.  However, this could mean your users who are dependent on that switch would be down for a minimum of a day.

Branch Wide Area Network (WAN) communications is another area to focus on redundancy.  A branch down is something we see fairly often in the Network Operations Center (NOC).  There are a couple options that I will cover and the first is duplicate circuits.  This option is probably the most expensive and I would only consider this if the particular branch in question is mission critical to operations.  This would give you the ability to have a router preconfigured to failover to another WAN circuit.

Another less expensive route would be utilizing your Internet connection and a firewall’s capability to build a Virtual Private Network (VPN) tunnel for branch communications.  This would require you to have an Internet connection that would adequately support all branch communication needs.  Internet would have to available at both branches with a firewall at each location.  Then, the firewall would have to be configured to handle branch traffic and routes would be configured in the WAN router to failover to the firewall if a circuit went down.

Finally, and arguably the most appropriate option, would be to build a network at your branch that could operate independently.  You may be asking yourself, “What does he mean by independently?”  I’m referring specifically to the ability of a branch to operate without any need for a connection to another branch.  Often, FIs will install core circuits, Internet connections, and banking applications locally at a branch so that in the event of WAN outage, the branch would continue to operate.

All of these areas previously mentioned can be adequately redundant but all are contingent on power.  If your FI has a power outage, all means to prevent an operational bottleneck will be thwarted.  Uninterruptible Power Supplies (UPS) are one way to mitigate this risk.  However, these are meant for temporary outages and will only address power outages based on the life of the batteries in the particular UPS model.  I recommend keeping spare batteries available as we often see these fail in FIs.  Most important are the UPSs used for communications and servers.  However, also consider a workstation UPS and keeping replacement batteries onsite for these models as well.

The best option, but most expensive, would be a generator.  A generator would be most appropriate at your main operations center and this would be a solution to address any long term power outages.  I suggest you consider this option if you are in an area that has frequent inclement weather conditions that result in power outages for your FI.  Furthermore, it is worth considering having a branch with an onsite generator – the branch could also serve as your Disaster Recovery (DR) site.

Lastly, duplicating equipment and planning for failure is not cheap.  However, I suggest you begin exploring your network and documenting areas that have single points of failure.  Then, I recommend you communicate these needs to your executive management team so that these needs can be addressed with the Board of Directors.  A great way to address single points of failure would be during your Business Continuity Planning (BCP).  During this process you will perform a Business Impact Analysis (BIA) that will help you address recovery objectives and provide a means to communicate the areas that are most critical to banking operations.  Keep in mind, Safe Systems’ Compliance Guru and all our compliance experts are available to assist with this pertinent process.

As always, thank you for your business and for your time in reading this article.

Write a Comment