Tuesday, 7 January 2014

Jmeter integration with Jenkin

Jmeter integration with Jenkin

Integration of jmeter with jenkin for performance testing on nightly build.

Following are configuration steps:

  1. Add the Performance Plugin.
  2. Click the Manage Jenkins link.
  3. Click the Manage Plugins link.
  4. Select the Available tab.
  5. Find the Performance Plugin, select it, then install it.

Scheduling job in jenkin:

  1. Go to the Jenkins main page.
  2. Click New Job.
  3. Give the job a name. test1
  4. Choose free-style software project.
  5. Click OK
  6. Under Advanced Project Options give the directory path of you framework as workspace. give it  some display name
  7. Add build step - Execute windows batch command
  8. Enter the command as  
    • perl newIn.pl -i SSOinputs.xml -c config.properties
  9. Add post build action Publish performance test result
    • Add jmeter report
Start the build schedule from jenkins home page.

An report will be email to the recipient.

Friday, 22 November 2013

Test setup for automation in python selenium

Welcome to my selenium blog.  I will be posting a variety of topics focused on selenium test automation.
This blog will show you step by step installation of selenium automation using Python
Python programming language is used to automate functional tests using Selenium WebDriver.
Step 1 : To run selenium test cases in python, you need python on your machine. You can download Python 3.3 installer from
The python-3.3.2.msi setup is 19.5 MB  in size.
Step 2 : You need to import different python libraries in your project. PyPI (Python Package Index) is a repository of software for the Python programming language.Setup tool is require to install the python bindings.

You can install setuptools by downloading ez_setup.py script.
Create easySetupTool directory under python installer. Then copy the script on your machine. 

Run the command:
C:\Python33\easySetupTool>python ez_setup.py

Once installation is complete, you will find an easy_install.exe program in your Python Scripts sub directory. Add this directory to your PATH environment variable.

You can also use pip command to download any package from python binding.
Step 3 : You can download Python bindings for Selenium from the PyPI page for selenium package.
The selenium package is used to automate web browser interaction with python.
Download it from here .
Step 4 : Create folder name selenium c:\selenium.
Step 5 : Unzip selenium webdriver and extract  "selenium-2.37.2"  in selenium folder.

Step 6 : Run setup.py script on command prompt as given below.
python C:\Python33\selenium-2.37.2\setup.py install

Instead of step 3, 4,5 and 6 you can directly download selenium package as:
run the command 
C:\Python33\Scripts>easy_install selenium
The setup is ready now. You can verify it by running any test script captured using selenium IDE.
Record any testcase using Selenium IDE and then export the test case as python 2.0 selenium webdriver. Save the script as test.py python script.
Run the test script as :
D:\PythonScripts>python test.py
C:\Python33\lib\site-packages\selenium-2.37.2-py3.3.egg\selenium\webdriver\firefox\firefox_binary.py:95: ResourceWarning: unclosed <socket.socket object, fd=284, family=2, type=1, proto
  while not utils.is_connectable(self.profile.port):
Ran 1 test in 51.239s
Ok indicates, your test is run successfully.

Wednesday, 20 November 2013

Activities involved during performance testing

In this post we will see different activities involved during performance testing of any application.

Activities performed-

  1. Tool Selection
  2. Create user scenarios
  3. Scripting
  4. Dry run of the application.
  5. Running a load test and analyzing the result

Tool Selection -

There are different tools used to measure performance like JMap (Heap memory usage, Creating memory dumps), JHat (Analyze the memory dump), JStack (Thread count), JConsole (Heap memory - graphical) , JMeter (API response), EhCache monitor (To see cached objects), PerfMon (Client side CPU, memory), YourKit (Server side CPU, memory, thread count) , Simple commands like top, free (Linux) and task manager (windows)

Depending upon the type of application and language support look for any open source or commercial performance tools convenient for your application

I have used jmeter for web based application implemented in java to measure API response time and EnCache monitor to check if cache is hit or not for whatever request.

User Scenario -

The test scenarios for performance test are based on different setup that needs to be tested. Hence as per the setup the test scenarios are executed:
  1. What is anticipated average number of users ?
  2. What is anticipated peak number of users ?
  3. How does application manage cookies, session-rewriting or any other method ?
  4. What is the testing intended to achieve?

Scripting -

There is a requirement to replicate the real scenario of production system. The data loading process is is a repetitive task for populating database.This include test setup preparation or creation i.e. per-requisite for running the performance tests and to load heavy data into the database.
These activities are automated with scripts.

Scripts are require for post test cleanup to restore back to original state for another performance run.

Dry run of the application -

Once the setup is ready it gives a brief insight into your application performance. Execute the performance run to obtain the initial performance result. Studying the results, decide the component to keep focus on. Also measure the performance result with different number of threads, ramp-up time, consider latency and tomcat configuration.
In all testing, response time should be ideally below a capped benchmark to particular values.
Dry run is necessary to setup ideal configuration of server and client. Final Performance results are carried on this setup. 

Running a load test and analyzing the result -

After executing a load test we need to interpret the result. For result interpretation we use Listeners that provide different metrics about the load test.

Performance suite should not be run only once it might give an inaccurate reading. For example, some requests fires multiple queries internally. The first request is going to be slow because a database hasn’t cached it yet. After results are cached every following request will take significantly less time. So the results are calculated as an average of three runs.

To know more about performance result analysis read my post on performance test result analysis read it here.

Analyzing Performance result

There are different parameters need to be consider during performance testing like average response time, throughput and latency. Test results are used for analyzing failed call rates.

The most important parameter is to find the response time for number of virtual users (like 50,100,200 etc.)

Jmeter is an open source tool designed to measure performance. It is use to simulate a heavy load on a server.There are different listeners available in jmeter to capture performance result. The most commonly used are Result tree and Summary report.The summary report gives information about average response time,standard deviation and error rate.Result tree can be used for debugging any request. It gives real time details about thread actions.

Performance results are analyzed in steps.Initially performance of all API requests per functionality for one users is executed.Then gradually scale the number of users. Cover the performance of different functionalities independently. eg. Login.

Measure the performance of each components used in product. Then determine bottleneck.

The performance result varies with log level. The response time is calculated by keeping the log level on ERROR.

Performance result also depends on tomcat performance.
A tomcat with default configuration can handle max 300req/sec. By tweaking tomcat connection parameters performance can be improved.

Consider the performance of database used in product. Make sure that all queries are appropriately used in code.

Parameters that are monitored during the performance test are as follows:
  1. Average Response time
  2. Standard deviation
  3. Harmonic mean
  4. Throughput
  5. Latency
  6. Request passed / failed
  7. CPU usage
  8. Memory usage
  9. Disk usage
  10. SQL queries to the database

Performance suite should not be run only once it might give an inaccurate reading. For example, some requests fires multiple queries internally. The first request is going to be slow because a database hasn’t cached it yet. After results are cached every following request will take significantly less time. So the results are calculated as an average of three runs.

Test setup for performance testing

Performance testing is carried out when the product is stable and ready for deployment on production.
Test environment has major impact on performance test result. Setting up the correct environment is important for effective performance testing. Test setup is configured for testing the java web application.

The test environment includes hardware, software, and network configurations on test instance.


To execute the performance testing it should be executed on server machines with higher configuration such as RAM,Memory and CPU architecture.Test machine should support latest version of performance test tool identified for load simulation.


Following list of software to run performance testing using Jmeter:
  • JRE - Java 6 or above.
  • apache-jmeter-2.7.z01.
  • apache-ant-1.8.2
  • ant - ant-jmeter-1.0.9.jar.
  • mail.jar
  • Apache tomcat 7.0.40


Identify the application type and its component. All the components require for successful test execution should be reachable and in same subnet.If its not feasible consider the network latency.
All simulated request fired from client should reach to server.

A dedicated test environments is require for data load generation and test execution. Any other services,scripts, cron jobs or application should not be running on the test machine. If there is any stale process or thread is available it will affect the memory utlization and other performance parameter.

When performance test is executed HTTP requests are sent to Apache tomcat and then it forwards the request to application server hosted on amazon EC2 instance. Application reads the data from RDS Mysql instance and then send back the response.

The test setup components are described in details as follows:
Server – Amazon CentOS instance with RDS mysql database.
Client - Windows-07 with Apache jmeter installed , SSO module running on Tomcat server.

Monday, 18 November 2013

Identity Bridge

This article explains new concept in Identity Management called Identity Bridge.
Identity Management solution is provided as a new cloud service known as IDaaS.
Identity Bridge is technology used in IDaaS solution.

Need of Identity Bridge -

Enterprise organizations already have existing user stores with their identities.A new architectural component is needed to manage the flow of identity information between cooperating organizations.
Enterprise identity stores and cloud identity stores are two mutually exclusive services that do not communicate, forcing organizations to spend double the time managing and aligning two disparate data sources. 

Challenges -

Most companies have on- premise IAM solutions which makes it difficult for software-as-a-service (SaaS) providers offering hosted services.

Organization want to utlize their on premise Active directory asset to manage cloud applications. There are two challenges while integration with organization active directory to make identity management simple and secure
  1. Employees must be provisioned into the SaaS application at administrative time.
  2. The business must authenticate employees and securely transition users from their enterprise login into the SaaS session at runtime.

Moving identity management to the cloud involves risks of auditing, ensuring compliance of regulations and what happens if disclosures occur. There is no insight into how Identity is managed in cloud.
Identity Management Process were previously behind a firewall and most likely always inside the network become exposed to the Internet via IdaaS.

What is an Identity Bridge?

Identity Bridge is a communication medium between enterprise identity store and cloud identity store. Identity Bridge is introduced to securely extend corporate identity beyond company firewall so users can seamlessly log on to cloud services - and has control over its access. It allows to manage identities for adoption and transition to cloud services while leveraging current investment in identity and access management technologies.
Identity Bridge accelerates cloud service adoption by automating the addition of new users and streamlining the onboarding process.

Identity Bridge is a new emerging technology in IDaaS where identity is managed by thirdparty resources .Outsourcing the most critical function of business to third party is not secure. Identity Bridge is an on-premise appliance can be hosted in cloud, enterprise premise or at service provider. Its a new customizable on-premises appliance offered to cloud service providers to seamlessly integrate the user identities of their enterprise customers with the cloud services.

Identity Bridge Services - 

Federation -

Identity federation provides secure authentication for cloud services. Identity bridge offers Idp and SP federation. It transform security tokens from a standard accepted in one realm (e.g., Kerberos tickets in an AD environment) to a standard accepted in another realm (e.g., SAML tokens in a web service environment or OAuth tokens in a mobile environment). federated single sign-on to recognize users across company systems, and real-time synchronization of user identities, making end-to-end user management across enterprise, cloud, and mobile environments instantaneous.

Directory synchronization – 

It provides various features for user sync between two disapparate entities. Identity synchronization is done from the identity provider (e.g., company’s AD implementation) to the service provider (the target cloud service) to ensure that changes made to the identity provider, such as disabling an account, are immediately replicated to the service provider.

Just-in-Time(JIT) provisioning -

A JIT provisioning ensures an account is created at the service provider only when a user first attempts to access the service. however, that JIT provisioning covers only the creation of the account.

Authorization (AuthZ) services

It determine who can access which services.

Password vaulting -  

Software as a Service (SaaS) providers doesnot set up to support federation. Instead, they rely on inputting a user ID and password to authenticate users to their service. Password vaulting stores a user’s credentials in the identity service and retrieves it to the SaaS website as if the user was directly logging on.


User provisioning refers to the creation, maintenance and deactivation of user objects and user attributes, as they exist in one or more systems, directories or applications, in response to automated or interactive business processes.

Auditing and Reporting -

Provide audit trail of user activity such as user and group authentication data and federation events across enterprise, cloud and mobile environments to meet compliance reporting requirements.

In addition, the bridge leverages caching, automation and transformation to ensure an efficient use of information and resources.

Identity Bridge Use Cases

To the Cloud -

Organizations that want to extend their existing identity management processes to manage users in SaaS or partner applications.
eg. Larger established companies which have significant on-premises IT infrastructure.

In the Cloud -

Organizations that want off-premise Identity management solutions for users and applications in the cloud.
eg. Smaller organizations whose core IT functions are delivered via SaaS applications. Or
Larger organizations for a specific user population.

From the Cloud -

Organizations want to leverage off-premises IDaaS for on-premises identities and applications. Many organizations aren’t comfortable yet with storing user information in an IDaaS application. A hybrid solution that stores user information on-premises.

Examples -

McAfee CloudSSO Identity Bridge
ForgeRock Bridge
Ping IdentityBridge