Pen test or penetration testing, may be defined as an attempt to evaluate the security of an IT infrastructure by simulating a cyber-attack against computer system to exploit vulnerabilities. What is the difference between vulnerability scanning and penetration testing? Vulnerability scanning simply identifies the noted vulnerabilities and penetration testing, as mentioned earlier, is an attempt to exploit vulnerabilities.
Penetration testing helps to determine whether unauthorized access or any other malicious activity is possible in the system. We can perform penetration testing for servers, web applications, wireless networks, mobile devices and any other potential point of exposure using manual or automated technologies.
Penetration testing is one strategy that can be used to mitigate the risks of cyber-attacks. The success of penetration testing depends upon an efficient & consistent assessment methodology. Using a methodology is that it allows assessors to evaluate an environment consistently.
Listed below are some of the methodologies:
- Open Source Security Testing Methodology Manual (OSSTMM)
- Open Web Application Security Project (OWASP)
- National Institute of Standards and Technology (NIST)
- Penetration Testing Execution Standard (PTES)
Why program when scanners are available?
There are a number of vulnerability scanners available in the market which can be used for vulnerability discovery, however such scanners have their own limitations and sometimes are not able to provide full coverage. This makes the job of a penetration tester a little more difficult, since a lot of things are missed out – especially if the product is complex. This is where custom scripts/tools come into the picture. They help in filling the gaps created by the scanner since they’re customized to the target application rather than being general purpose.
Scanners tend to perform poorly when the target application is very complex thus leaving out many pages of the applications uncovered, and hence the false negatives – which create a far more challenging situation for the security team, and the management which completely rely on scanners for automating the vulnerability assessment. False positives are easy as the security analyst can easily eliminate them. However, false negatives are difficult since these are the issues which go unreported.
It should be noted here that custom tools written for specialized purpose using languages like python should not be a replacement for vulnerability scanners, and ideally should be used in addition to these scanners to get the best throughput.
Let’s focus on modules that can be used for making custom HTTP requests using python.
Setting up the Environment
Installers are available for Python and can be downloaded from
– http://www.python.org/getit/
Windows users can download the installer from above mentioned URL and install Python. To further make the use of python easier, Windows users can add python to the system path by editing the environment variable. Once done, users can just fire up python from the command prompt – irrespective of the current working directory and still be able to invoke python interpreter.
If you are a Linux or Macintosh user, chances are high that you don’t have to install python, since it comes pre-installed for Macintosh by Apple and by concerned vendor for most Linux distros To check if Python is installed on your system, launch the command prompt and type “python”, if python is pre-installed, we should be able to see the interpreter getting launched as shown in the following screen shot:
If not follow the below steps (kali Linux users):
- sudo apt-get install -y python3-pip
- sudo apt-get install build-essential libssl-dev libffi-dev python-dev
We first need to install the venv module, part of the standard Python 3 library, so that we can invoke the pyvenv command which will create virtual environments for us. Let’s install venv by typing:
- sudo apt-get install -y python3-venv
Let’s choose which directory we would like to put our Python programming environments in, or we can create a new directory with mkdir, as in:
- mkdir environments
- cd environments
Once you are in the directory where you would like the environments to live, you can create an environment by running the following command:
- pyvenv my_env
Essentially, pyvenv sets up a new directory that contains a few items which we can view with the ls command:
- ls my_env
Output
bin include lib lib64 pyvenv.cfg
You’ll find this in each of the lib directories.
To use this environment, you need to activate it, by the following command that calls the activate script:
- source my_env/bin/activate
Your prompt will now be prefixed with the name of your environment, in this case it is called my_env.
- (my_env) root@rockstardevil:~/environments#
Your virtual environment is now ready to use.
Python Modules for crafting HTTP Requests
Python has multiple modules that can be used for generating custom HTTP Requests. We’ll cover 2 such modules that can be used for developing customized scripts, and can fire up our payloads along with performing the same actions that a penetration tester performs manually – the only difference being, this is done by a script instead of a manual attempt. The modules that we will cover are:
httplib
This module has been renamed to httplib.client in python 3, however we will be using httplib version2.7.* as far as this article is concerned. Normally this module is not directly used but instead urllib module uses it internally to make HTTP Requests.Interested users can always use it directly.
In order for us to send custom requests, we need to follow the following steps:
1. Import the library
Before using a library, we need to import it. As we are going to use httplib library to send HTTP Requests and receive the responses back, we need to import it.
>2. Create a Connection
Once imported, we can start using it straight away. In this step, we need to create a connection object first. This can be achieved using HTTPConnection() method
3. Send HTTP Request
So far no HTTP Request is sent on the wire. In order to do so, use request() method. This is when the HTTP packet that we have created in previous steps is sent out over the network to the target web server using the method passed on as an argument (in our case GET, as shown in following figure).
4. Get HTTP Response
Now that we have sent a request, we can use getresponse() object to get server’s response. This method will return HTTP Response object back, which when read will send output generated by the server.
The following Screenshot lists these four steps executed from an interpreter:
In our case, invoking a HTTP request is resulting into a 302 redirect. It should be noted here that we can’t read “res” variable as it is, since it will only point us to the HTTP Response object. The read () method provides a raw string which is returned by the target web server as shown in above figure.
urllib2
“urllib2” is a little different from “httplib” library when it comes to creating and sending out HTTP Requests. “urllib2”. We don’t have to open up a connection and instead after importing, we can directly make a request –as shown in following figure:
This is much simpler when compared to httplib. It is suggested that users make use of urllib2 as it’s recommended even by the python community.
It’s recommended that readers go through the python documentation to understand what all functions are supported by urllib2 module to explore full potential of this library and utilize it when creating your own tools or scripts.
In the following section, we’ll see a sample SQL Injection tool that I’ve created only for demonstration purpose. It hits the login page of the website and injects single payload.
Sample SQL Injection Script:
The following is a simple script:
import urllib
import urllib2
location = "http://test_target.site/login.aspx"
values = {"username":"'","password":"password","btnSubmit":"Login"}
data = urllib.urlencode(values)
req = urllib2.Request(location,data)
response = urllib2.urlopen(req)
page_data = response.read()
print page_data
First we are importing urllib and urllib2 libraries. We then associate the target URL to variable “location” and assigned post data to variable “values”. Once these steps are completed, we are encoding the URL data and then submitting the request to the server and reading the response received.
The above script is just to show how easily one can create custom tools. The above script is far from perfect and will need much modification before using in practice. It only fires one request, while in real life our tool should fire multiple requests by iterating over a list of payloads. It’s left as an exercise to readers to go through libraries and the functions it supports to understand how they can create their own tools. A real life tool will also have to take care of session management and hence needs to also deal with cookies and other HTTP headers like referrers, Content Type etc. We’ll also need to iterate over the a list of URL’s repeatedly until all our payloads are fired one by one for each and every parameter in order to ensure coverage.
