Comparisons with industry standard frameworks and Limitations
Learning Curve
Benefit of using this tool is that learning curve is very small and doesn’t require any programming skills to start. This immediately helps in including in the testing process, the CJEs, Business users, Platform and other DevOps Engineers who do not have programming or test automation background.
Readymade Framework
Selenium, Appium, HttpURLConnection, WinAppDriver are just libraries, not testing tools. A framework needs to be created from scratch or reused from open source market, to use these libraries.
This tool does all heavy lifting with readily available framework developed
on top of "Selenium", "Appium", "HttpURLConnection", "WinAppDriver"
Comparitive Analysis
It is important to note that in the comparison below, we will only include popular java or python frameworks and not libraries like Selenium, Playwright etc.
Comparing frameworks with libraries is like comparing apples and oranges!
The reason to choose java/python and not javascript is that, in Wholesale Banking, most applications are Packages and Commercial off-the-shelf (COTS) and monoliths. These applications do not have modern Web UI and are heavily focussed on the Business Processing side of things. Javascript based tools and frameworks are primarily aimed to test Modern Web UI. With Java we have a greater edge to test the processes like, Java Messaging Queue, Kafka Streams, Reports (xlsx, pdf), DataBase etc. in addition to Web and API.
Features |
CITS |
Cucumber |
Serenity |
Robot Framework |
Karate |
Katalon |
Test Complete |
|---|---|---|---|---|---|---|---|
Open Source |
✓ | ✓ | ✓ | ✓ | ✓ | Freemium | ✗ |
Integrated IDE with data sheets and property files |
✓ | ✗ | ✗ | ✗ | ✗ | ✓ | ✓ |
OS Independent |
✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✗ |
Shallow Learning Curve |
✓ | ✗ | ✗ | ✓ | ✓ | ✓ | ✓ |
Community Support |
✗ (Community is growing but still small) | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
High Ease of test case creation |
✓ | ✗ | ✗ | ✓ | ✓ | ✓ | ✓ |
Business User Friendly |
✓ | ✗ | ✗ | ✗ | ✗ | ✓ | ✓ |
Built-in Record and Spy for quick start |
✓ | ✗ | ✗ | ✗ | ✗ | ✓ | ✓ |
Cross Browser Testing |
✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
Multiple Object Properties in Object Repository |
✓ | ✗ | ✗ | ✗ | ✗ | ✗ | ✓ |
Distributed Execution |
✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
Integration with Selenium Grid |
✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
Built-in support for BDD approach |
✓ | ✓ | ✓ | ✗ | ✗ (Not to confuse with Karate BDD like DSL) | ✓ (Plugins needed) | ✓ (Plugins needed) |
Built-in support for Data-driven approach |
✓ | ✗ | ✗ | ✗ | ✗ | ✓ | ✓ |
Built-in support for Keyword-driven approach |
✓ | ✗ | ✗ | ✓ | ✗ | ✓ | ✓ |
Customizations and enhancements |
✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
Built-in Native Azure DevOps reports with logs and screenshots |
✓ | ✗ (Junit doesn't support screenshots) | ✗ (Junit doesn't support screenshots) | ✗ (Plugins not supported in ING) | ✗ (Junit doesn't support screenshots) | ✗ (Junit doesn't support screenshots) | ✗ (Plugins not supported in ING) |
Built-in Azure DevOps Test Plan Integration |
✓ | ✗ | ✗ | ✗ | ✗ | ✗ (Plugins not supported in ING) | ✗ (Plugins not supported in ING) |
Built-in API Testing |
✓ | ✓ (additional libraries needed) | ✓ (additional libraries needed) | ✓ (additional libraries needed) | ✓ | ✓ | ✗ |
Built-in Web Page Performance Testing |
✓ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ |
Built-in Accessibility Testing |
✓ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ |
Built-in Database Testing |
✓ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ |
Built-in Image Based Testing |
✓ | ✗ | ✗ | ✗ | ✗ | ✗ | ✓ |
Docker Support |
✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
Limitations of the tool
-
CITS is a tester-friendly tool. Developers, who are accustomed to only coding and writing scripts in an IDE like IntelliJ or Eclipse might find the UI of the tool, limiting.
-
Even though the tool has an exhaustive Java project in the backend which can be customized, enhanced and even tailored for project specific purposes, it is not the first thing that a developer sees, thereby creating a feeling of lack of coding liberty.
-
While CITS is designed to be user-friendly, there is still a learning curve involved in using the tool effectively. One may need to invest time in learning how to use the tool's features and scripting capabilities to create effective tests.
-
The tool's community is quite small. Even though it is growing and efforts are being put in to create exhaustive documentation and videos, it will take some time for it to become popular.