ThreatDown Safe Deployment Practices

At ThreatDown, we are reimagining security to safeguard devices, data, and people, no matter where they are located, how they work and play, or their budget. Our suite of award-winning cybersecurity and privacy solutions empowers millions of people worldwide to enjoy their digital lives, free from threats and scams. To deliver on this promise, we use safe deployment practices that create resilient, repeatable processes for updating and managing our solutions.

Overview

Malwarebytes has a robust quality assurance process throughout the deployment cycle that involves:

  • Extensive code coverage and static code analysis
  • Compatibility, functionality, performance, and system impact testing
  • Staggered and phased releases
  • In-house and third-party monitoring systems and utilities
  • Direct customer feedback
  • Rollback and kill switch mechanisms

We take pride in being a global leader in cybersecurity, with a responsibility to protect our customers’ digital lives and ensure a seamless experience. Read on to dive deeper into our process.

Thorough Development & Testing

To start, ThreatDown follows the best practices in software development with checks and testing embedded at every stage of our development cycle. For example:

  • Code Development: We perform code coverage and static code analysis with both manual and automated security checks, followed by an intensive inspection of the code for vulnerabilities and other potential issues that might impact stability, security, and overall usability of the ThreatDown product.
  • Testing: We conduct extensive testing across the product lifecycle to confirm compatibility, functionality, performance, and system impact. For example, we test across multiple Windows operating system versions and hardware configurations in x86, x64, and ARM architecture, including new, existing, and dark features (feature flags). Performance and system impact tests are also conducted to guarantee that security does not create usability issues for customers.

Feedback-driven Phased Deployment

All product updates, including enhancements, security or protection updates, and new features, are gradually deployed to users via a phased deployment (or metered) process. Each deployment is closely observed and completed before additional waves are deployed. We focus on clearly defined success metrics during this process to balance any urgency of need for the update with maintaining product functionality.

The phased development process is driven by a mixture of both telemetry-driven and user-based feedback with clear actions for every possible scenario that may occur during deployment. Disciplined schedules, entry and exit gates based on deployment percentage, and cross-referencing with what is seen in deployment monitoring and customer feedback.

Monitoring and Key Deployment Indicators

Using data-driven telemetry and clearly defined actions for every possible scenario, we leverage both in-house and 3rd-party monitoring systems and utilities, multiple indicators are referenced to identify:

  • When critical problems occur, such as crashes, incompatibilities, and performance-dragging incidents
  • Features that are not functioning as expected – in both the service and the UI level of the application
  • Protection and detection technologies are creating customer inconveniences, such as false positives and/or false negatives

Apart from the direct-from-devices Key Deployment Indicators, there is also a next-level analysis of the data to identify anomalies, which are addressed with proper mitigation and often become automated, thus becoming key deployment indicators.

Customer Engagement

Each deployment cycle is powered by the guidance of customer feedback. This starts with engaging customers with the list of features and capabilities of the new software update

through Betas, the ThreatDown website, forums, customer success channels, and active engagement in social media such as Facebook, Instagram, Reddit, etc.

We assist customers regardless of where they say, “I need some help”, and social media is one of the best platforms for deployment feedback. We also offer 24/7 global support so customers can reach out to us regardless of time zone or location.

Recovery and issue mitigation

At ThreatDown, we understand that even with our thorough safe deployment practice, deployment issues may arise. We have established mitigation procedures depending on the problem encountered.

If the problem is caused by a detection or protection issue and it affects a very small portion of the customer base, a recovery lever that can be utilized is a single detection fix using rapid response mechanisms. For larger scale impacts, detection technology whiteouts or protection update rollbacks are employed to immediately put customers back in a stable, secure state. Meanwhile ThreatDown triages and starts a cycle of updates to mitigate the problem.

For feature or protection layer incidents, “killswitches” built in to disable features are often tapped for mitigation. Otherwise, an application “roll forward“ solution based on the previous stable build is quickly prepared and rolled out to all affected customers, restoring stability without sacrificing security and protection of the devices. All these mitigation capabilities are fully documented, mostly automated, and regularly tested with both announced and unannounced drills at least once every quarter.