Unfurl: An Entropy-Based Link Vulnerability Analysis

Forensic study of internet based crime has not been easy always due to limited tools for acquisition and analyzing of data.

Internet has been a source of huge amount of data in form of web pages and files which can be uniquely identified by URL. Analysis of URL and its hierarchy plays important role to obtain domain and other information related to webpages.

Unfurl analyses a URL and creates a directed graph by extracting every bit of information from the URL and exposing the obscured. URL is broken up into small components, by extracting as much information as it can from a piece. This approach makes the analysis transparent to the user and helps them learn about semantic and syntactical URL structures.

 

It also has some more generic parsers (timestamps, UUIDs, base64, etc.) helpful for exploring new URLs or reverse engineering. No matter if you extracted a URL from a memory image, carved it from slack space, or pulled it from a browser’s history file, Unfurl can help you get the most out of it.

Features:

1. Unfurl has parsers for URLs from popular search engines, mail services, and chat   applications.
2. It is also easy to build new parsers.
3. Unfurl is open source (Python 3) and has an extensible plugin system.
4. Remediation of the remote system if access is authorized by the owner.
5. Remote scanning of other network nodes (useful for remote incident response.
6. Source code of this tool is available free.

How to use Unfurl:

 1. Online Version

1. There is an online version at https://dfir.blog/unfurl/.Visit that page, enter the URL in the form, and click 'Unfurl!’.
2. You can also access the online version using a bookmarklet - create a new bookmark and paste “javascript:window.location.href='https://dfir.blog/unfurl/?url='+window.location.href;”       as the location. Then when visiting any page with an interesting URL, you can click the bookmarklet to the URL "unfurled".

  2. System Install

1. Clone or download Unfurl from GitHub.
2. Install Python 3 and the modules in requirements.txt
3. Run python unfurl_app.py
4. Browse to localhost:5000/ (editable via config file)
5. Enter the URL to unfurl in the form, and 'Unfurl!'

 3. Docker version

1.   git clone https://github.com/obsidianforensics/unfurl
2.  cd unfurl
3.  Modify unfurl.ini with desired host and port, and docker-compose.yaml to match port defined in  unfurl.ini.
4. docker-compose up -d

 4. For Testing

1.  All tests are run automatically on each PR by Travis CI. Tests need to pass before merging
2. While not required, it is strongly encouraged to add tests that cover any new features in a PR.
3. To manually run all tests (units and integration): python -m unittest discover -s tests.

 

Now unfurl can also expand analyse onion url’s as shown above that is a great invention in forensic analysis of Deep web.

  

Parsing Unknown Protobufs with Unfurl

Protobufs are quite common on mobile devices and I suspect there are more in URLs than the few known ones in Google Search.

If you haven't read Yogesh's post (you should), a key concept about protobufs is that they are minimal; you have to make guesses when parsing them if you don't know the original data type for each item.

With the latest update, Unfurl can now parse protobufs as well! It's using slightly-modified blackboxprotobuf code, so the "assumptions" it makes about the data before displaying are the same. Here is the same tester_pb being parsed with Unfurl.

Unfurl isn't designed to take files as input, so you can't just drop a protobuf file in it. It supports reading protobufs three ways (right now):

• as a hex-string (example above),
• as standard base64-encoded (example above),
• and as URL-safe base64-encoded (example above).

It means that every node that looks like a potential protobuf will get tested as one.We hope to discover more interesting data in URLs this way. If you Unfurl a URL and find some protobuf data, please let us know!