Posts

Prepare Your Business for the Next Normal

(Updated 5/4/20)

With some states and local jurisdictions beginning to loosen or remove stay-at-home and essential business orders and advisories, many small businesses will begin to adjust for the next phase of response and recovery.  For some, this will be a re-opening; for others it will be another shift in how we conduct our business on a day-to-day basis.  Either way, the process will be a minefield of financial, operational, legal, liability, and personnel issues. Before “flipping” the sign from closed to open, plan your return with care and compassion. Both will be needed to keep your employees, customers, and business safe.

Prepare the Groundwork

Guidance on opening is coming from many sources. We recommend a top-down approach, starting at the federal level and working down the your local municipalities and property owners.

  1. Start with the expertise and guidance from the US Centers for Disease Control and Prevention (CDC).  The CDC website  provides guidance for different types of businesses and gathering places that centers on three mitigation strategies:
    • Personal protective measures (e.g., hand-washing, cough etiquette, and face coverings) that persons can use at home or while in community settings
    • Social distancing (e.g., maintaining physical distance between persons in community settings and staying at home)
    • Environmental surface cleaning at home and in community settings, such as schools or workplaces.
  2. Review current laws and regulations under the Families First Coronavirus Recovery Act (FFCRA). This legislation requires almost all employers to provide expanded sick time, medical leave, and family leave pay for employees dealing with illness or childcare issues themselves or within their immediate family unit.  Make sure your return to work plans accommodate these programs and
  3. Second, understand your state’s rules and regulations with with respect to physically opening your business.  Many states are staging how they will allow business to open.  Then, check with local governments where your business is located and where your employees live.  In some states, municipalities and counties are adjusting how they implement state and federal orders and advisories to address local needs and issues.
  4.  Understand your state’s unemployment rules and regulations. In some states, lifting of stay-at-home orders may mean employees are no longer eligible for unemployment even if you keep your business closed or cannot bring everyone back to work. Your team will have differing concerns and levels of comfort; it is important to provide them with timely and accurate communications.
  5. Check with your landlord. Many office and retail complexes are setting up guidelines and rules for how businesses can and will be able to operate in their properties.  Some office complexes, for example, are limiting access to employees only and restricting access to trades and delivery personnel.
  6. Ask your landlord what additional steps they will be taking to clean and sanitize bathrooms, elevators, stair railings, door handles, and other common areas and high touch surfaces.  You and your employees will want and need to know how safe the environment will be when then return to the office or store.

With an understanding of how you can and want to take your next steps, create a Communications Plan.  More than just determined who, when, and how you will share information with employees and other stakeholders, the plan should provide a clear and easy way for employees to get answers to their questions.  As many smaller businesses do not have internal HR resources, you may want to assign a particular manager or executive team to the role.  If you have a contracted HR service or consultant, you will need to coordinate both the process and information. Set clear expectations for how quickly you will answer questions and how answers to common questions will be addressed to the company at large.

Prepare Your Place

As you do your groundwork, begin planning and putting your workplace together for the return of staff.  Social distancing is the current normal. With an expected recurrence of COVID-19 in the fall, social distancing will be part of our lives, and work places, for some time to come. For employees to return, you may be considering:

  • Setting up protocols to ensure that workers who may be ill, or have been exposed, do not enter the workplace and accidentally infect others.
  • Placing dividers between work spaces, or re-configuring your office layout to create separation.
  • Acquiring additional office space, temporarily, to allow more team members to return.
  • Requiring the use of masks or other appropriate personal protective equipment (PPE). Depending on your work environment, this may be full-time or only when employees leave personal work spaces and head to common or communal areas.
  • Cleaning and sanitation of common areas, like kitchens and break rooms, and high touch surfaces.
  • Coordinating disinfection and sanitation efforts with building management and neighboring businesses in leased office spaces.
  • Ensuring availability of cleaning supplies, disinfectants, and sanitizers.
  • Creating a means for employees to express concerns about the work environment and actions of others, without fear of retribution.

For some businesses, the safest course of action will be establishing split shifts or a rotating schedule of employee teams working in the office. Doing so can ease physical separation issues, but we should expect that some employees will need to, or want to, continue working from home.

Prepare Your People

Communications — timely, open, and honest — will be critical for successfully taking the next steps with your business.  For many, personal anxiety and stress will be high as we navigate shifts in our personal and work lives.

Provide your team as much information as possible on what to expect, and how things will move forward, as you go through each upcoming phase of your plans.

As you communicate with your team, keep in mind that employees may be dealing with personal COVID-19 impacts, such as:

  • Death of a family member of close friend
  • Sick or quarantined family member(s)
  • Loss of income by a spouse/partner/family member
  • Supervision of children learning from home
  • Lack of available daycare
  • Anxiety and stress
  • Feeling unable to return to working in the office

Be prepared to deal with the human side of Covid-19, not just the logistics.

  • Anticipate and have answers ready for employees about your requirements and their options
  • Establish a feedback loop and listen to staff issues and concerns
  • Engage your HR staff, service, or consultants to assist with communications, feedback, and responses
  • Update plans and timing as needed to mitigate staff concerns and business conditions

Prepare to Settle In

Set Expectations

As noted, above, experts are telling us to expect local/regional COVID-19 outbreaks throughout the fall and winter. With this expectation, we should plan for future stay-at-home orders and business restrictions. These will likely vary by location, complicating your planning efforts.

Remote work will be part of our operations for the foreseeable future. As you plan your next steps, make sure that your team is ideally equipped to continue working from home.

In the scramble to respond to stay-at-home orders, many businesses make necessary technology decisions for the near-term.  Now is the time to step back and take a long-term view. Employees may be working on home computers, using personal software, and working in a less-then-ideal space. Many businesses are also finding employees have signed up for free or consumer IT services to work around limitations, such as difficulty accessing files on company servers.  We still have a responsibility to keep information secure and private, and our employees and businesses safe.

Get Your IT Resources in Place

Settling in means adapting work environments — at the office and in employees’ homes — to our anticipated reality.

  • Improve security and access to company systems and data
    • Move data from on-premise servers to cloud file services to improve access and security; Map drives to cloud-data for compatibility with desktop software
    • Use Remote desktop and VDI solutions to move on-premise applications to the cloud, providing easy, high performance access without distributing data to remote computers
  • Ensure employees have workable use of your phone system (see this post for more info)
  • Reduce the need for remote PC, VPN and other remote access solutions that increase cost, complexity, and delays
  • Eliminate the need for shadow IT services by helping employees use existing capabilities in your productivity suite
  • Provide devices for employees that do not usually work from home
    • Consider rental, lease, and device-as-a-service option to manage costs
  • If unable to provide devices, upgrade home computers:
    • Add memory for performance and ensure the ability to run business applications
    • Deploy licenses of business software, even if employees are using consumer versions of the applications
    • “Next Gen” endpoint protections from viruses, malware, and ransomware
    • Web filtering and DNS security to prevent malware from infected websites
  • Provide employees with helpful accessories, such as noise cancelling headsets for video calls

We are here to help you plan and execute your next steps.  Our free Response and Recovery Assessment will help you with your planning, fully utilize your existing IT Services, and identify budget-friendly solutions to address any unmet needs and priorities. Email us or complete the form on our home page to schedule your assessment.


 

Coronavirus: Prep Instead of Panic

Updated Mar. 23, 2020. (new content in italics)

Coronavirus is in the news with broad localized impact.  The Centers for Disease Control is urging everyone to prepare for a major public outbreak of COVID-19 will strike the United States. Here is a high level update:

  • Extensive “Community Outbreaks” have been identified by the Centers for Disease Control (CDC), with confirmed cases rising at a sharp rate in several areas of the country.
  • Several states, counties, and local governments are issuing lock-down or “shelter-in-place” orders, severely restricting business and personal activities.
  • Many states and local governments restricting meetings and services.
  • School are closing for extended periods of time. Government facilities are closing to the public
  • Employers are restricting travel and meetings
  • Employers are telling employees to work from home
  • Restaurants and bars are restricted to carry-out and delivery only
  • Mandatory quarantines are in place in some areas

What does this mean for us and our businesses?

Experts currently agree that potential for wide-spread outbreaks exists and there is an urgent and critical need to minimize the spread of the virus.  That said, we should expect, and be prepared, to address localized issues.  These can include:

  • Employees being quarantined at home, or while traveling, due to possible exposure or systems.
  • Closing offices to facilitate social distancing
  • School closings, requiring employees to remain home with children.
  • Localized building shutdowns, including government offices, courts, etc.
  • Impact on, or hesitancy to use, public transportation; reduced public transportation schedules
  • Cancellation of conferences, meetings, and events.
  • Hesitancy to travel, or restrictions on travel destinations.

Each of these feels manageable if the inconvenience is only for a limited time. But with quarantines running 14 days (or more) and concerns that the virus might live on surfaces for as long as nine days, these disruptions may create serious challenges.

Question to Ask and Consider

  • Have we communicated a policy to employees that “Safe is better than Sorry”
    • Do employees know to stay home and avoid meetings if they are not feeling well?
    • Should your business alter or halt normal operations?
    • Should you close your facilities to some or all employees?
  • Can your employees easily and efficiently work remotely?
    • Does this include employees who normally work at the office?
    • What materials, documents, or services might they need?
    • Will functions, such as customer service, function properly with remote users?
    • For employees that don’t normally work at home:
      • Do they have sufficient Internet bandwidth?
      • Do they have the software needed to use your VoIP phone service?
      • Do they have an appropriate device (personal or company-provided)?
      • If using personal devices, do they have the necessary software and versions?
    • Can you extend business phone service to workers at home? If so, are you comfortable
      getting this setup?
  • Do you have the ability to replace travel and on-site meetings with video conferencing?
    • Are enough of your conference rooms equipped for conferencing?
    • Is your team comfortable using the equipment and services?
    • For employees who do not normally work remotely or use audio/video conferencing:
      • Do they have access to audio/video conferencing services from their devices?
      • Do they understand how to use the services from their computers or phones?
      • Are they comfortable with using these services?
  • What functions can be scaled back or delayed with minimal impact to operations, cash flow, customer service, etc.?
  • Which functions are critical to your business continuity?
  • Will supply chain issues disrupt your business?
  • How might the evolving economic fallout impact your business?

Steps to Take

As you consider and answer the above questions, and others, you can better understand how to prepare.  For some, enabling more remote work may be as simple as a temporary cloud file service or migrating files from on-premise file servers to cloud file services.  Other businesses might consider ensuring team members have suitable computers at home or company laptops available if needed. Now might be the time to add Teams Meeting or Hangout Meet hardware and services to your conference rooms and huddle areas. And some businesses may want to expand remote access to business systems or relocate applications to cloud servers.

The good news is that you are not alone.  As you monitor events, assess your risks, and plan, we are here to help.

Please:

  • Join our Open Office Hours on Wednesday March 18, 2020 at 2:00 pm ET.
  • Contact us and take advantage of our expertise and, if appropriate, our services.

We are in this together.

Thank you,

 

 

Allen Falcon, CEO and Pragmatic Evangelist

Business Continuity

Risk and Reward – Protecting the Value of Your Business

Business ContinuitySeveral weeks ago, in a town not far from our headquarters, a massive fire destroyed a building housing six small businesses.  Our local business journal followed up a few weeks after the disaster with a poll asking business owners how prepared they are for a major disaster.

  • Fewer than 50% of responding business owners feel that they are fully insured, have an emergency plan, and could be up and running in a few days.
  • 39% feel that it could take a month or so, but they could eventually reopen
  • 17% felt they would be out of business or would required state and local aid to survive

While not a scientific sampling, the results are alarming.  Alarming for a few reasons:

  • Even with insurance, it can take days or weeks to get authorization so you can move forward with your emergency plan.  Securing a new location and replacing fixtures, inventory, etc. takes time, as does recovering computer systems and data.
  • More than 50% of businesses closed for 7 days due to a disaster fail within 6 months of reopening.  While many businesses might re-open in a month, the future will be challenging.

Your Risks are Yours

A major fire in a block of retail and service businesses creates specific challenges, as do storms and floods.  Many more businesses, however, experience disasters equal or greater in scope even if they do not have the same level of physical damage. Some examples we have seen.

  • A distributor of customized office supplies lost all electronic business records for the past three years when they where hit by ransomware. The attack corrupted their on-site backup servers as well as their main file and database servers.
  • A news publisher lost all of their physical servers, firewalls, and networking equipment when a sprinkler head failed in their small equipment room.
  • A small plastics manufacturer lost the ability to use their process control systems when embedded Windows workstations were corrupted by a malware attack.

In each of these examples, businesses with customer commitments, production schedules, and deadlines were idled for days. For some, full recovery can take months.  Beyond the hard cost of recovering systems and data, these businesses suffered from soft cost losses.  Missed customer commitments, delayed invoicing and collections, and the time employees spent on the recovery effort all have lasting impacts on your business.

Business Continuity is a not just a good idea, it is a responsibility. 

As business owners, our employees, vendors, and customers count on us.  While people can empathize with the impact of a fire, there is less understanding for businesses that fall victim to cyber crime.  Malware, phishing, ransomware and other attacks are generally preventable when your team is alert and aware of the risks and when you put reasonable identity, data, and system protections in place. And since no protection is perfect, you need to be able to recover quickly enough for your business to continue operating smoothly.

Here is some food for thought:

  • Know Your RTO:  Understand how quickly your business needs to Return to Operational.  Maybe you can work on paper for a few days. Maybe you need to be up and running in a few hours because you are at a standstill until systems are back online. Your RTO goal will guide your decisions on what protection and recovery/continuity services are the right match for your needs and budget.
  • Assess Your Risk: Understand the different disaster scenarios and how they may impact your business.  Think about physical issues, such as loss of power and catastrophic system failures, as well as other disruptions, such as cyber attacks and potential actions by a disgruntled employee.
  • Watch Your Flank: Asses how different types of threats could impact your business.  We are beyond hiding our computers behind firewalls. We still have physical threats, but we also have threats focused on networks, user identities, access control, third party services, and data sources and services. Each threat vector needs a plan for protection, response, and recovery.
  • Factor in Humanity: We used to talk about balancing security with ease of use.  Today, the humanity equation is different as most IT disasters take advantage of human factors like our fundamental desire be helpful when asked. In many ways, your team is your best defense. They need to understand the risks, the methods of manipulation, and the signs that something is not quite “right”.  Your team needs to understand the value of inconveniences like multi-factor authentication and enhanced privacy and access controls — that these protect them as well as the company.

Your next step.

Contact us.  It is time for a serious conversation about protecting the value of your business.  A basic assessment of your business continuity profile will identify risks and gaps. From there, we can discuss improvements and their business value so you can make informed decisions that balance your risks, needs, and budget.  Business Continuity solutions — from disaster prevention through recovery — do not need to bust your budget.   For most business, changes in security settings on existing systems paired with modest, incremental services provide the protection and recover-ability you need.

Newtons Cradle

Inertia: The Science of Business Continuity

Newtons CradleTo paraphrase Newton’s Laws of Motion (with credit to Galileo) …

Absent an unbalanced force, an object in motion will stay in motion and an object at rest will stay at rest.

While this holds true for objects in a friction-less environment, it holds true for our businesses as well. Our businesses are in motion, working each day to service our customers with rhythms and cycles throughout each day, week, month, and year.

Our business cycles continue, until we meet an unbalanced force.

Some forces we expect, like changes in the economy that occur over a period of weeks or months.  Others forces are event-driven, such as storms, cyber attacks, and key employee departures. The sudden nature of event-driven forces can catch us by surprise, cripple our businesses in the short-term, and disrupt our normal cycles for the long-term.

A Case in Point

A company here in the northeast manufactures and distributes a customized product that customers generally replace or re-order every 2 to 3 years.  80% of the firm’s business is repeat, creating a strong and stable business. The company was hit by ransomware twice in a 3 month period.  The first attack, scrambled their files and their servers, but left their financial system in place.  They lost a day’s worth of data.  The immediate recovery took 3 days; the full recovery took nearly two weeks.  After three days of cleaning systems and restoring data, the company’s systems were up and running. They then had to enter the initial day lost data and all of the business activity for the 3 days their systems were down.  They allocated 1/3 of everybody’s time to recover the data, reducing productivity by 33% and impacting their responsiveness to customers. To enter the 4 days of missing data took over 10 days with the team working part time.

Inertia Takes Hold

This initial event changed the cycles and motions of the company. Whenever dealing with any business activity during the outage and recovery periods, they need to double check to make sure the information entered was complete and correct. And since some activities, like shipping and invoices related to prior activities, they need to double-check these connections.  Long after the two week recovery period, productivity is still down as the company’s daily motion now includes double-checking information that they are not sure they can trust.

Lesson NOT Learned

With so much focus on getting the business back into its normal rhythm, and the additional cost involved, the company did not act on recommendations that could help prevent a future attack and better ensure their ability to recover should a future attack occur. Whether the second attack was a different attack or they had failed to fully clean their systems does not matter.  The second attack was not caught until after the company’s backup server was hit, rendering their backups useless.  The company lost three years of data.

Inertia Creates a New Cycle

To recover from this attack took more than balancing data entry and on-going business. It was not feasible to manually recreate three years of data. While entering about 6 months of data for the fiscal year, they settled for a solution that created new methods and rhythms with long-term effects. They recalled all of their paper records from storage into an expanded warehouse space.  When a customer calls to re-order product they ordered 2 or 3 years ago, they search and retrieve the physical paperwork so they can create the new order. Every returning customer creates a scramble to find the paperwork in short order. Actions required in an emergency become part of the new normal. Inertia.

What You Can Do

You can be prepared with solutions that balance external forces beyond your control.

  • An educated and aware workforce balances the human manipulation that enables cyber attacks
  • Advanced threat, DNS, and web protections balance the forces of cyber attacks hitting us daily.
  • A robust backup/recovery and continuity system balances the forceful impact of disruptive events, giving you the ability to be up and running in hours not days.

If the company in our case study had implemented the recommended solutions after the first attack, they second attack would have disrupted the business for less than half a day — and may not have happened at all. The investment in communication, prevention, and recovery, while not trivial, was minor compared to the short term recovery and long term impact on the business.

If you are not ready or willing to have your business’ inertia redirected by forces beyond your control, now is the time to act.


Contact us for a free, no obligation, Cloud Advisor Session to discuss your business recovery and continuity needs and plans.


 

Pending Storm; Pending Doom

A quick scan of the weather headlines late on Thursday afternoon: a “Nor’easter” storm going through rapid escalation, know as “Bombogenisis”, looks ready to hit New England tomorrow with rain, snow and hurricane force wind gusts. Now it is Sunday, and many small and midsize businesses along the northeastern coast are wondering when, or if, they will be able to reopen. The impact of disasters is increasing. We can argue about climate change versus weather. We can discuss our aging infrastructure. We can debate whether to plan for disaster causes or effects. If we do not, however, make our businesses more resilient, the quantity and severity of disruptions will continue to grow.

The coming storm should not foretell coming doom.

By taking advantage of proven cloud services, most small and midsize businesses can protect themselves from disruption. Many businesses in coastal areas of New England may be without power and other utilities for 2 to 4 days. Businesses with no continuity plan are down and out. Given that about 50% of businesses shut down for a week will fail within six months, “down and out” can be fatal. If you rely on VPN or remote desktop to on-premise systems, you are still at risk — no power means no on-premise networks or servers.

Businesses with key systems in the cloud, however, can be up and running if employees have power and Internet access.

So what are your next steps?

First, measure the impact on your business of a disruption lasting one day, three days, and five days?  As you do, consider the full cost of recovery, including post-disaster productivity loss as your work to recover lost data and time while keeping things moving forward.

Second, consider the value of keeping your business running rather than having to recover and regroup. Beyond the dollars and cents, understand the value to your customers, to your reputation.

Third, contact us for a complimentary Cloud Advisor Session to discuss your cloud and continuity strategies.

Overconfidence in Disaster Recovery: Common and Costly

support-liferingAs reported in CloudTech, a recent study in the UK of 250 businesses finds that 95% experienced outages or data loss in the past year, with 87% needing to go to failover systems.

There is a mismatch between expectation and reality when it comes to disaster recovery.

Of the 87% that executed a failover, 82% were confident it would go well, but 55% encountered problems. And while 69% stated outages lasting minutes would be “highly disruptive” or “catastrophic”, only 27% were able to recover all systems immediately following an outage. With 37% of respondents indicating they do not regularly test their DR capabilities, many organizations have no basis for expecting a smooth failover.

Outage Sources

While we often focus on the “big disaster” that could interrupt our businesses, 53% of the outages were to mundane system failures and 52% were due to human error (more than one response was possible). Cyber attacks and environmental issues caused 32% and 20% of the outages, respectively.

Three Things We Can Learn

  1. Comprehensive disaster recovery and business continuity costs money. Running infrastructure and systems in the cloud and/or using cloud-based DR and Business Continuity solutions can help mitigate these costs.  You will, however, need to assess potential downtime and time to recover, the impact of downtime, and the cost to create the right balance for your organization.
  2. Testing your DR/Business Continuity solutions should be easy and cost-effective. Plan on testing at least twice per year.
  3. Your DR/Business continuity solution should help reinforce your overall data protection and business operations. Shifting from a “recovery”-centric strategy to one of resilience can lower costs and minimize the risks and impacts of unplanned outages.

If you want to improve your business’ resilience and lower your IT costs, contact us for a free Cloud Advisor session.


 

Rethinking Risks and Responses

Malware, Ransomware, Natural Disasters and More Keep Hitting SMBs Hard

Never have we had a greater ability to work together to get things done than we do right now. As our cloud and hybrid environments expand, the ease-of-use encourages us to share ideas and information and to collaborate in new and exciting ways.

Never have we been under attack from so many directions. Changing weather patterns and aging infrastructure leave businesses without power for days instead of hours. Fading employee loyalty means more chances for information to walk out the door. The same features that let us easily share information also let us accidentally share information we shouldn’t. Malware and viruses have evolved from a nuisance to potentially existential threats with the increase in ransomware and advanced persistent threats.

Our Businesses, Employees, and Customers Need and Expect Protection

With the risks and impacts on the rise, we as small and midsize business owners and technologists should rethink how we both prepare and respond. Since the dawn of business computing, large enterprises have built expensive solutions to ensure that their businesses keep running “no matter what”.  Now that we are in the cloud, and solutions are incredibly affordable, we need to adopt the same approach.

Business continuity is no longer just being able to keep your business running after a disaster.

Business continuity means that you are able to prevent business disruptions and distractions, regardless of the cause. Business continuity means …

  • You actively work to minimize the chance of a ransomware attack, and that you can respond and recover quickly should it happen.
  • You have systems and procedures in place to prevent data loss and privacy breaches, and that you can detect and mitigate issues quickly and effectively.
  • You and your team are no longer tethered to the hardware, Internet access, and electricity in your offices.

For SMBs, now is the time to consider the tangible and intangible costs of business interruptions of all types and to see the value in solutions to prevent and recovery. Understand the value proposition of that goes beyond dollars and cents to include the customer relationship impact and the toll that business disruption has on your team.

Food for Thought:

Calm Before the Storm: 3 Models for Protecting Your Business

Hurricane
What began as a mild tropical storm season has suddenly become quite active, with multiple significant storms expected to impact the southeast and Atlantic coast and the Hawaiian islands. And while every storm may not be a major hurricane, your business is at risk because our infrastructure is at risk.

Power outages, local or regional flooding, and disruption of communication services continue to increase in frequency as our infrastructure ages faster than our upgrades and as our economy rewards utilities for trimming staff and services rather than trimming trees and keeping current with maintenance.

Are you protecting your business from the damage and risk of disruption?

You have seemingly infinite choices on the types and cost of protection, each with benefits and limitations. Your challenge: pick the solution that is most cost-effective, meaning the time it takes to Return to Operations (RTO) is acceptable given the cost.

To simplify your search for a solution, we propose you consider one of three models:

  • Restoration
  • Recovery
  • Continuity

Restoration

Restoration is the least expensive option.  You backup all of your data and critical systems, including full system images, off-site.  In the event of a disaster, you restore your systems once you have fixed or replaced any damaged or lost equipment.

  • Cost Structure:
    • Scales with the size of your system images and the amount of data you keep in offsite backup
  • RTO:
    • 1 to 3 days once replacement equipment arrives
  • Admin:
    • Must ensure backups include all images and data needed to recovery, including Bare Metal Restore (BMR) for key servers and systems.
    • Must periodically test restore for data integrity and to ensure the recovery process is documented and understood.

Recovery

In addition to keeping an off-site or cloud backup covering all of your data and critical systems, you have the ability to access replicas of your network and servers in a remote data center.  In the event of a disaster, you “spin up” your latest system snapshots and restore any incremental data. You access your mirror network via remote desktop, VPN client, or LAN-to-LAN VPN.

  • Cost Structure:
    • Scales with the size of your system images and the amount of data you keep in offsite backup
  • RTO:
    • 1 to 18 hours, depending on your configuration and needs.
  • Admin:
    • Must ensure backups include all images and data needed to recovery, including Bare Metal Restore (BMR) for key servers and systems.
    • Must periodically test recovery for data integrity and to ensure the recovery process is documented and understood.
    • Once primary systems are repaired or replaced, snapshot backups and recovery move your data back for normal operations.

Continuity

Continuity means your IT infrastructure keeps running, even in the face of disaster or significant local events.  You have multiple options for continuity, including: mirrored networks and systems in remote data centers, remote desktops, virtual desktop infrastructure (VDI), and Desktop-as-a-Service (DaaS) models. In each scenario, your servers, applications, and data live in a redundant, remote cloud data center. You access your environment via remote connection, using a web browser or a small local app known as a receiver.  In the event of an emergency, you only need to provide a browser and Internet connection to be up and running.

  • Cost Structure:
    • Scales with the size of your systems and networks
    • Offsets day to day costs of owning and managing on-premise hardware and software
  • RTO:
    • Immediate, based on Internet availability
  • Admin:
    • Providers typically include standard server admin and management, reducing local need for IT resources
    • Application and data management are similar to on-premise systems
    • Backup/restore capabilities are still recommended to protect against application and/or human error.

Using these models as a guide, you can select a solution that balances cost, convenience, and complexity against the operational needs of your business.


Want to setup or improve your disaster recovery/business continuity capabilities? Contact us for a free, no-obligation consultation.


 

When the Single Point of Failure Actually Fails

 

While the heavy, wet snow continues to fall and cling to the power and fiASA5505ber optic lines in our area, today’s Internet outage was not due to the first real storm this winter. Being fully in the cloud, any Internet outage could be a disaster, bringing business to a halt. In reality, the “single point of failure” really isn’t. True, we do not have multiple routers. Nor do we have multiple broadband connections.  What we do have, is the ability to work over any form of Internet connection. Here is our case study (still in progress).

Late yesterday afternoon, our trusted Cisco 5505 stopped working. Poof. Red Status light on; activity lights on the embedded switch ports blinking; no traffic. A few reboots and a few attempted hard resets later, we are still not working. A quick call and discussion, and our Cisco guru tells us “it’s a brick”. Covered by warranty and a solid support/service plan, a new unit will arrive in several days. In the meantime, we must continue to service our customers.

Quick Fix

The immediate response is to get our staff connected to the Internet in any way possible.  A few mobile hotspots activated on our phones and one MiFi device booted up, and we are back in business. Performance is acceptable, not great, and we will plow through our data plan, but we are in business with only a few minutes disruption.

Interim Fix

Our FiOS service enters our office through a service unit that converts the Fiber to Gigabit Ethernet.  We split this signal through a switch to 2 routers — one provided by our VoiP service and the FiOS router/cable modem that comes with our service.  The now dead Cisco ASA plugs into the FiOS router.

Why two routers in sequence? Having 2 routers in sequence creates a physical DMZ: a network that can receive traffic from inside and the outside while letting us stop traffic from going all the way out or coming all the way in. It’s “old school” as virtual DMZs are the trend.  We use the DMZ and the FiOS router for a guest network and wireless.  Guests can gain access to a physical or wireless connection while staying completely outside our secure network. The Cisco ASA, at the secure end of the DMZ, manages our inbound traffic, NAT, and legacy DMZ services (let over from the days when we had a few systems on-premise and needed remote access). Our secure WiFi runs off a Cisco/Linksys WAP inside the secure border of the ASA router.

With a few minutes of work, we reconfigured the FiOS router, removing the DMZ and mimicking the settings and security configured in the ASA.  Moving a few wires, we are up and running until the new ASA comes in.

Lessons Learned

Our focus has always been on the FiOS service as the single point of failure at greatest risk.  Outages have traditionally been short and as we have been able to adapt by using hotspots, MiFi, and working from home or other locations, we have not seen the need to bring in another ISP as an alternate service. The ASA failing was never really a consideration.  The box is not yet out of warranty and our prior Cisco routers lasted much longer than the 5 year extended warranty (we upgraded for features, not out of necessity).

Not having seen this scenario coming, we had to rebuild the FiOS router from scratch. Going forward, we have now saved this “emergency configuration” for future use.  Once our new Cisco ASA arrives, we will create an emergency configuration that will let us remove the FiOS router from the network.  Finally, we will build a configuration for the Cisco/Linksys WAP, as this has routing features and could replace the FiOS router in a pinch.

The biggest lesson, however, is the value of a cloud-based infrastructure with respect to business continuity. Storm or no storm, hardware failure or not, we know that we will always have options to keep our business up and running. Even when the “single point of failure” happens to fail.

Resilience Trumps Continuity

business resilience
The unexpected will happen. It is inevitable. Sometimes the unexpected is a good thing. In technology, the unexpected is usually bad.  It may be small … or big … or catastrophic.

Part of our role as IT professionals is to expect and prepare for the unexpected. We backup data so that we can restore files that are accidentally deleted, overwritten, or damaged.  We backup systems so that we can recover them in case of hardware or software failures. Many business design and implement disaster recovery plans. These plans provide the means for companies to recovery from larger incidents, ranging from burst pipes and building fires to blizzards and hurricanes.

In recent years, the focus has been on “Business Continuity” planning. Business continuity intends to prevent disruption to operations, even in the face of larger incidents or disasters. While great in concept, most small and mid-size enterprises cannot afford to fully duplicate systems in redundant data centers and provide alternate work sites for employees.

Enter Business Resiliency!

Business Resiliency is based on the objective of enabling a business to continue (or rapidly resume) operations with some accommodations.  In other words, you may not be running 100%, but you will be running soon enough and well enough given the situation. Resiliency is about bending without breaking.

Consider Hurricane Sandy that devastated parts of the US Eastern Seaboard.  Many businesses were physically destroyed by the flooding. Many others were shut down by the indirect effects of the flooding as some areas along the coast lost critical infrastructure — including water and sewer. Businesses left physically intact but without power for days considered themselves lucky as some areas waited months for reconstruction.

Consider the ice storms and blizzards throughout the Northeast US in recent years.  For many businesses, the only disruption was loss of power.  And while in many of the storms outages where generally localized, some businesses went without power for as long as three weeks.

The same holds true for businesses in “tornado alley” in the midwest. A tornado may leave your business unscathed, but it may take days or weeks for power and water to be restored.

In each of these scenarios, backup/restore/recovery is not enough to get the business back up and running. And, again, most small and mid-size businesses cannot afford to maintain disaster recovery systems and sites.

Cloud Fosters Resiliency!

Most businesses can afford to move IT systems into cloud computing and hosted solutions.  And in doing so, businesses can affordably build resiliency.

With all of these disasters, you did not have to travel too far inland to be out of the damage zone.  Businesses with on-premise equipment had to purchase and wait for delivery of replacements, rebuild their systems, and (hopefully) recover their data from their off-site backups. Certainly doable, but costly and time consuming.  It can take 2 to 4 days just to get the equipment in place and ready to restore.

Businesses in the cloud faced a different scenario and outcome. Moving to an area with power and Internet, businesses running in the cloud were up and running in hours (some in minutes) and some were never “down” at all.


To discuss how cloud computing can improve the resiliency of your business, contact us for a no-obligation conversation or click here to learn about our RestartIT solutions.