We created a basic crawler, but now, we want to make it recursive to cover all the web application content.
- View Offline
- [Voiceover] In the previous video,…we learned how to create a basic web application crawler.…In this video, we will be adding recursion capabilities.…In this video, we're going to start learning…how to extract links.…Then we're going to use them to make the crawler recursive.…Now that we have created the basic structure of our crawler,…let's add some functionality.…First, let's copy the prepared spiderman.py file…for this exercise.…Copy from Examples/Spiders/Spiderman-recursive.py to…basic_crawler/basic_crawler/…spiders/spiderman.py.…
Then we go back to our editor.…As we would like to make the crawler recursive,…for this purpose we will work again in the spiderman.py file…and we will start with adding another extractor…but this time, the links, instead of titles, like here.…Also, we need to make sure that the links…are valid and complete.…So we're going to create a regular expression…that will validate links seen here.…This regular expression should validate…all HTTP and HTTPS absolute links.…
Now that we have the code to extract the links,…
Stop using automated testing tools. Customize and write your own tests with Python! While there are an increasing number of sophisticated ready-made tools to scan systems for vulnerabilities, Python allows testers to write system-specific scripts—or alter and extend existing testing tools—to find, exploit, and record as many security weaknesses as possible. This course will give you the necessary skills to write custom tools for different scenarios and modify existing Python tools to suit your application's needs.
Christian Martorella starts off by providing an overview of the web application penetration testing process and the tools the professionals use to perform these tests. Next he shows how to interact with web applications using Python, HTTP, and the Requests library. Then follow the web application penetration testing methodology. Each section contains practical Python examples. To finish off, Christian shows how to use the tools against a vulnerable web application created specifically for this course.
- Understanding web penetration testing
- Interacting with web applications via HTTP and the Requests library
- Analyzing HTTP responses
- Web crawling with Scrapy
- Extracting information
- Discovering resources
- Testing passwords
- Detecting and exploiting SQL injection vulnerabilities
- Intercepting HTTP requests