Safe Deployment Practices at ThreatDown
At ThreatDown, we are reimagining security to safeguard devices, data, and people, no matter where they are located, how they work and play, or their budget. Our suite of award- winning cybersecurity and privacy solutions empower millions of people worldwide to enjoy their digital lives, free from threats and scams. To deliver on this promise, we use safe deployment practices that create resilient, repeatable processes for updating and managing our solutions.
Overview
ThreatDown has a robust quality assurance process throughout the deployment cycle that involves:
- Extensive code coverage and static code analysis
- Compatibility, functionality, performance, and system impact testing
- Staggered and phased releases
- In-house and third-party monitoring systems and utilities
- Direct customer feedback
- Rollback and kill switch mechanisms
We take pride in being a global leader in cybersecurity, with a responsibility to protect our customers’ digital lives and to ensure a seamless experience. Read on to dive deeper into our process.
Thorough Development & Testing
To start, ThreatDown follows the best practices in software development with checks and testing embedded at every stage of our development cycle. For example:
- Code Development: We perform code coverage and static code analysis with both manual and automated security checks, followed by an intensive inspection of the code for vulnerabilities and other potential issues that might impact stability, security, and overall usability of the ThreatDown product.
- Testing: We conduct extensive testing across the product lifecycle to confirm compatibility, functionality, performance and system impact. For example, we test across multiple Windows operating system versions and hardware configuration in x86, x64, and ARM architecture, including new, existing, and dark features (feature flags). Performance and system impact tests are also conducted to guarantee that security does not create usability issues for customers.
Feedback-driven Phased Deployment
All product updates, including enhancements, security or protection updates, and new features, are gradually deployed to users via a phased deployment (or metered) process. Each deployment is closely observed and completed before additional waves are deployed. We focus on clearly defined metrics of success during this process to balance any urgency of need for the update with product functionality.
The phased development process is driven by a mixture of both telemetry-driven and user-based feedback with clear actions for every possible scenario that may occur during deployment. Disciplined schedules, entry and exit gates based on deployment percentage, and cross referencing with what is seen in deployment monitoring and customer feedback.
Monitoring and Key Deployment Indicators
Using data-driven telemetry and clearly defined actions for every possible scenario, we leverage both in-house and 3rd-party monitoring systems and utilities, multiple indicators are referenced to identify:
- When critical problems occur such as crashes, incompatibilities, and performance dragging incidents
- Features that are not functioning as expected – in both the service and the UI level of the application
- Protection and detection technologies are creating customer inconveniences, such as false positives and/or false negatives
Apart from the direct-from-devices Key Deployment Indicators, there is also a next-level analysis of the data to identify anomalies, which are addressed with proper mitigation and often become automated, thus becoming key deployment indicators.
Customer Engagement
Each deployment cycle is powered by the guidance of customer feedback. This starts with engaging customers with the list of features and capabilities of the new software update through Betas, the ThreatDown website, forums, customer success channels, and active engagement in social media such as Facebook, Instagram, Reddit, etc.
We assist customers regardless of where they say, “I need some help”, and social media is one of the best platforms for deployment feedback. We also offer 24/7 global support so customers can reach out to us regardless of time zone or location.
Recovery and issue mitigation
At ThreatDown, we understand that even with our thorough safe deployment practice, deployment issues may arise. We have established mitigation procedures depending on the problem encountered.
If the problem is caused by a detection or protection issue and it affects a very small portion of the customer base, a recovery lever that can be utilized is a single detection fix using rapid response mechanisms. For larger scale impacts, detection technology whiteouts or protection update rollbacks are employed to immediately put customers back in a stable, secure state while ThreatDown triages, and starts a cycle of updates that mitigate these issues.
For feature or protection layer incidents, “killswitches” built-in to disable features are often tapped for mitigation. Otherwise, an application “roll forward“ solution based on the previous stable build is quickly prepared and rolled out to all affected customers, restoring stability without sacrificing security and protection of the devices. All these mitigation capabilities are fully documented, mostly automated and regularly tested with both announced and un-announced drills at least once every quarter.