The Company You Keep
There is an old joke in the financial industry: “What do bankers do if their core provider goes down? They wait for it to come back up.” In today’s banking environment, this simply doesn’t cut it anymore. If a financial institution does not thoroughly analyze its vendors as part of the business continuity planning (BCP) process, it opens itself up to the risk of extended downtime. It is crucial for banks to know exactly how they are going to recover if their vendor goes down. Regulators expect (and mandate) that banks have alternative procedures and processes in place in the event of disruption of service from a mission-critical provider. As part of the BCP process, banks must determine a Recovery Time Objective (RTO) for every internal process. If a process depends on a vendor, then a disruptive event at that vendor impacts the bank’s ability to provide services. For this reason, banks must take vendor recovery capabilities into account when determining their own recovery procedures. Do you know the RTO’s for your critical vendors’ services?
A good example of such an event is the disruption that an item processing company experienced during Super Storm Sandy. A core provider had an item processing center on the East Coast with a replication site in Texas. They would periodically test their DR solution by cutting off the connection to the main site and switching over to the replication site. While their tests always ran smoothly, the provider did not test running operations from the replication site at normal full volume. When the primary processing center was flooded by the storm, a “real life” switch over was initiated, and the result was anything but smooth. Many banks were without item processing for 2-3 weeks. This was devastating to the financial institutions since they could no longer create cash letters, credit deposits to customer accounts, or even know definitively how much cash to have on hand. Normal business operations in a bank depend on being able to process items in a timely manner. Many believe this devastating event and resulting hardship is what brought about the updated FFIEC guidance.
What Can Financial Institutions Do to Assure Themselves That Their Vendors Are Doing Everything They Can to Recover?
Guidance requires that controls be in place to mitigate risks associated with third party vendors. While banks have no direct control over testing the third parties, banks must have a level of understanding or a high level of confidence that their third party service providers are doing everything they can to prepare for a disruption. While trust is important, banks should also verify that plans have been validated through testing and that the provider can recover within promised timeframes.
To achieve this, a financial institution should seek documentation from their vendors. Ideally, banks should obtain a third party attestation regarding the vendor’s internal controls, most commonly a SOC 2 report. This report should contain the data availability trust criteria, proving the independent audit firm that produced the SOC report reviewed the vendor’s practices to make sure systems can be effectively recovered. This is the primary report financial institutions should use to evaluate each vendor.
Not every critical provider has completed a SOC 2 audit. They may have only completed a SOC 1, which verifies that they are maintaining integrity of their financial reports but does not reveal anything about what they are doing to make sure services stay up and running. Furthermore, the type of the SOC 1 or SOC 2 report makes a difference in how effective the report is as a control. A “type I” report verifies the vendor’s controls are adequate, but does not include a testing component. A “type II” report extends the engagement to verify that the controls in place are not only adequate but are also validated through testing. Ideally, vendors should provide a SOC 2 Type II
In addition to the SOC 2, financial institutions should request the third party service provider’s BCP plan along with BCP testing results. Banks then must review these items and look for potential red flags and weaknesses — a complex process that is almost impossible for the average community bank to assess on their own. While it is a daunting task for a community bank to review a third party service provider’s test results, it really is an important element to the BCP. In the Super Storm Sandy example mentioned above, if the bank would have obtained the provider’s test results, they would have seen successful failover tests to their replication site year after year, but may also have recognized that testing was not to scale. Obtaining the proper documentation is important, but actually reviewing it and verifying that it addresses your concerns is an equally crucial step.
Disasters happen. Disruptions happen. As financial institutions increasingly leverage third party providers to support operations, they must take the time to properly assess those vendors’ own business continuity processes and ensure their ability to recover critical IT systems and resume normal business operations — regardless of whether the process is supported in-house or by a third-party service provider.