New Report: The Thin Red Line – Examining Penetration Testing Practices

Penetration testing is one of the oldest and most widely embraced disciplines within the field of cybersecurity.

Today, thousands of organizations, including large financial institutions and credit agencies, hospitals and insurance companies, military and intelligence agencies, and power and electric companies have placed their faith, their reputation, and the fate of our most valuable digital assets in the hands of penetration testers, or pentesters, as they are known.

Organizations that engage pentesters are looking for trouble. It’s the entire point of an exercise that seeks to identify weaknesses in an organization’s defense before it’s too late.

Clients hire pentesters to play the role of adversary in a simulated attack, and they often easily subvert expensive security solutions, imperil sensitive data, and phish or otherwise embarrass system administrators and executives – all with the permission of the client.

This report is meant to start a conversation and lift the veil on a range of pentesting practices, byproducts, and after effects about which clients and the general public may be unaware.


As the pentesting industry has evolved and expanded, the line distinguishing red teaming exercises (a military term that, for many, has come to be associated with services that include pentesting) from actual threat actor behavior has thinned and, in some cases, blurred entirely.

In the pages that follow, the BlackBerry Cylance Threat Intelligence Team examines the pentesting side of that thin red line.

Our study sheds light on a discipline where a lack of universally accepted standards allows a range of common practices that may be inadvertently introducing a host of hidden risks that could adversely impact the values, including client privacy and security, pentesting was intended to protect. These practices consequently raise critical questions about one of the fundamental paradigms of cybersecurity: the reduction of risk.

The research findings include:

  • A case study of one group identified and profiled by one security company as a so-called advanced persistent threat group (APT) which our research found is, in actuality, operating openly as a Brazilian security company offering pentesting services.
  • Evidence suggesting that while this company was respected by some clients, it was also likely responsible for exfiltrating more than 200 megabytes of sensitive reconnaissance data for a country’s air traffic control system – data we later found in a semi-public malware repository.
  • The widespread exposure of client data in semi-public repositories impacting airports, healthcare organizations, major financial institutions, huge technology companies, state and local governments, global non-profits, big retailers, and U.S. federal government agencies – all revealed via malware, phishing lures, and command-and-control (C2) infrastructure created by more than two dozen pentesters whose tradecraft we examined.
  • New and disturbing questions about the pentesting industry’s compliance with client expectations of privacy and confidentiality, as well as new legal and regulatory requirements like Europe’s General Data Protection Regulation (GDPR).
  • A litany of pentester-generated suites of malware and other hacking tools designed for use by network defenders that are now in the hands of several nation-state and organized crime threat actors, detailing their use in a host of real attacks in ways that make identification and proper attribution of the attacker more difficult.
  • An overview of the practical, ethical, and legal guidelines available to pentesters and, a discussion of the implications of the lack of an industry-wide standard for clients.

The goal of this report is to provide a view of pentesting from the security researcher’s perspective in an attempt to better educate other researchers, pentesters, and, most importantly, the clients they both seek to serve. We will discuss the potential for negative outcomes from pentesting activity in the hopes of prompting a dialogue that will catalyze efforts to implement a commonly accepted set of standards for best practices in pentesting.


Guest blog courtesy of BlackBerry Cylance. Read more BlackBerry Cylance blogs here.