Most applications have resources that are not linked and tools such as crawlers or proxies won’t find. So we need to discover resources with other methods.
- View Offline
- [Instructor] Hi and welcome to section four, Resources Discovery. In the previous section we saw how to write our own crawler using Python and the scrapie library. In this section we're going to learn what Resource Discovery is and why it is useful when testing the web application. We're going to start by creating our own brute forcer with Python and requests library and then we will improve our brute forcer in order to get better results. Finally we're going to take a screen shot of the resources detected.
Let's get on with the first video. In this video we're going to learn what is Resource Discovery and why it is important when testing web applications. Also we're going to introduce Fuzz DB which is going to be used in the next videos as our dictionary database. Again remember in section one where we learned about penetration testing process? In the process the second phase was mapping. In the mapping phase we need to build a map or catalog of the application pages and functionalities.
In the previous sections we learned how to perform application mapping using a crawler. We also learned that crawlers have some limitations. For example, links generated with JS are not identified by crawlers. This could be overcome by using http proxys or using a headless browser like Phantom JS. Still if we use all those we're going to identify all the resources that are linked somewhere in the web application.
But my personal experience has shown me that we can find many resources that are not linked. In order to discover these we need to perform Resource Discovery via dictionaries of known words. These kind of tools are known as dictionary attacks. Basically, because we're going to use a list of known words in order to identify the resources. Brute forcing, because we are using brute force in order to identify resources when using lists of permutations or combinations of strings.
Fuzzing, not really correct, but many times used to refer to resource discovery. What can we find using this technique? Files, such as backup files, test files, notes, scripts, documentation and examples. Directories, such as admin interfaces, backups, internal areas, upload directories, actions. Whenever there are verb names in options or parameters we can use dictionary of similar words to identify other functionalities.
Servlets, which are similar to actions but with file. Parameters, we can enumerate ranges or combinations of potential valid strings used in parameters. In order to be successful when we are doing resource discovery you need to have good quality lists. There are many dictionary databases where you can find many word lists appropriate for different environments or scenarios. FuzzDB is one of the most used and complete databases available on the internet.
We are going to use it in our next video. For resource discovery we're going to focus on the predictable resource locations dictionary. I recommend you take a look at it in our virtual machine. Under the code samples for this section, get familiar with the different dictionaries or string lists available. In this video we learned what resource discovery is and why it is important when testing the security of a web application.
In the next video we'll start developing our own brute forcer, let's get ready.
Stop using automated testing tools. Customize and write your own tests with Python! While there are an increasing number of sophisticated ready-made tools to scan systems for vulnerabilities, Python allows testers to write system-specific scripts—or alter and extend existing testing tools—to find, exploit, and record as many security weaknesses as possible. This course will give you the necessary skills to write custom tools for different scenarios and modify existing Python tools to suit your application's needs.
Christian Martorella starts off by providing an overview of the web application penetration testing process and the tools the professionals use to perform these tests. Next he shows how to interact with web applications using Python, HTTP, and the Requests library. Then follow the web application penetration testing methodology. Each section contains practical Python examples. To finish off, Christian shows how to use the tools against a vulnerable web application created specifically for this course.
- Understanding web penetration testing
- Interacting with web applications via HTTP and the Requests library
- Analyzing HTTP responses
- Web crawling with Scrapy
- Extracting information
- Discovering resources
- Testing passwords
- Detecting and exploiting SQL injection vulnerabilities
- Intercepting HTTP requests