The Sherlock project aims to provide OSINT collectors with a tool to search out usernames across multiple platforms. This could be useful during a search for someone or something’s social media accounts, or to identify usernames that could be used for typosquatting attacks against your organization or client. Automated tools like Sherlock help expedite these time consuming, but sometimes necessary, activities.
How does it work
The Sherlock project’s developers have done a great job of documenting how their tool works, and how you could potentially add additional functionality to it. A quick search of the Sherlock project turns up two additional Github projects that expand the sites searched by the tool. You can find these two projects at the following two links: NSFW Data and Sherlock Data (we will cover this in a second Sherlock post soon). If you follow this Github link, it will take you to a subpage of their GitHub project. There it will explain how to add new sites for the tool to check and some limitations to consider before adding them. I found this to be very useful for both adding new sites to my checklist and understanding how the tool works.
As the developers thoroughly explain on their page, Sherlock tries to navigate to a URL of the site with the username included in it. For example: www[.]socialmediaplatform[.]com/username. This method only works if the platform of interest uses this structure within it’s site.
According to Sherlock’s documentation, it uses three detection methods to determine if a username exists:
- HTTP Status – If the page does not exist then an http status code would normally be generated. The developers consider this to be one of the most effective methods but caveats it with it doesn’t work for all sites.
- Response URL – The developers consider this the next most effective means of determining whether a username exists. In this case the tool attempts to navigate to the url, if it fails the site redirects the tool to a different url. Once the tool detects the redirect, it understands that the username does not exist.
- Error Message – Considered to be the least reliable means of the three, this method searchs for text on the response page. This method only works if the exact string designated remains unchanged. Any site updates or changes to the message could potentially throw off the tools ability to detect the message.
Sherlock requires Python 3.6 or higher
# clone the repo $ git clone https://github.com/sherlock-project/sherlock.git # change the working directory to sherlock $ cd sherlock # install python3 and python3-pip if they are not installed $ apt-get install python3
$ apt-get install python3-pip
# install the requirements $ python3 -m pip install -r requirements.txt
The Sherlock tool has a number of options you could utilize during your investigation including the ability to make requests via tor and a number of options to output your results in different formats. Below we have included the out put for Sherlock’s help documentation.
$ python3 sherlock --help usage: sherlock [-h] [--version] [--verbose] [--rank] [--folderoutput FOLDEROUTPUT] [--output OUTPUT] [--tor] [--unique-tor] [--csv] [--site SITE_NAME] [--proxy PROXY_URL] [--json JSON_FILE] [--timeout TIMEOUT] [--print-found] [--no-color] [--browse] USERNAMES [USERNAMES ...] Sherlock: Find Usernames Across Social Networks (Version 0.12.2) positional arguments: USERNAMES One or more usernames to check with social networks. optional arguments: -h, --help show this help message and exit --version Display version information and dependencies. --verbose, -v, -d, --debug Display extra debugging information and metrics. --rank, -r Present websites ordered by their Alexa.com global rank in popularity. --folderoutput FOLDEROUTPUT, -fo FOLDEROUTPUT If using multiple usernames, the output of the results will be saved to this folder. --output OUTPUT, -o OUTPUT If using single username, the output of the result will be saved to this file. --tor, -t Make requests over Tor; increases runtime; requires Tor to be installed and in system path. --unique-tor, -u Make requests over Tor with new Tor circuit after each request; increases runtime; requires Tor to be installed and in system path. --csv Create Comma-Separated Values (CSV) File. --site SITE_NAME Limit analysis to just the listed sites. Add multiple options to specify more than one site. --proxy PROXY_URL, -p PROXY_URL Make requests over a proxy. e.g. socks5://127.0.0.1:1080 --json JSON_FILE, -j JSON_FILE Load data from a JSON file or an online, valid, JSON file. --timeout TIMEOUT Time (in seconds) to wait for response to requests. Default timeout of 60.0s.A longer timeout will be more likely to get results from slow sites.On the other hand, this may cause a long delay to gather all results. --print-found Do not output sites where the username was not found. --no-color Don't color terminal output --browse, -b Browse to all results on default bowser.
Sherlock also supports single and multiple lookups.
A single lookup looks like:
$python3 sherlock username.
A multiple lookup would look like
$python3 sherlock username1 username2 username3.
Below I’ve provided a few screenshots running Sherlock against my favorite OSINT Missing Persons CTF non-profit, TraceLabs. In the first example, I decided to utilize the –print-found option to limit the returned items to just positive matches. You will notice that a few errors still show up, but for the most part it eliminates the extra noise of accounts it did not find.
Ideally, you would then go back through your results and verify the findings. Remember that the Sherlock tool is doing an exact search for your username, so variations would not be returned. Normally I would run some variations of the username to see if we could find similar accounts that have a slightly different makeup. This has proven fruitful for me on more than one investigation.
Overall I’ve found Sherlock to be a very useful tool during some of my investigations. More than once it has helped me find additional sites or platforms being used by a subject during a TraceLabs Missing Persons CTF (https://www.tracelabs.org/). The added functionality built into the command line also allows analysts to focus the tool and provide different outputs depending on their preferred formats. Furthermore, the ability to add new sites to the tools capability makes the tool that much more powerful for analysts with unique requirements. Sherlock should be in every investigators tool box.