The ICTestAutomation experience
Last few months I’ve had the pleasure of to try out a new test automation framework called ICTestAutomation.
It offers you to automate from an user (end-2-end) perspective using scriptless test automation, OCR and image recognition technology.
Since I’m a coder myself, I thought of it as yet another record and play tool. But it actually delivered great results in our challenges. This blogpost shares my thoughts and experience (including pros, cons, tips and tricks) about this great tool from Trendic.
The challenge and approach
My team and I were responsible to give insight on the quality of the upgraded workstations including software installations. We were talking about 9 different workstations with software applications like Firefox, Chrome, Microsoft Edge, Office, Adobe PDF reader and other client specific applications. We also needed to verify that the performance of these applications didn’t degrade after the workstation upgrades like hardware and operating systems.
Obviously I did not want to gather data manually. So I started exploring what ICTestAutomation had to offer. Even-though I’m used to automate tests with open source libraries, I was amazed by the effectiveness and efficiency of the tool.
First step was specifying the test objectives. We had several dialogs with key users of the organisation to identify the verification requirements and succescriteria for each objective.
Since we had to setup different testcases on different workstations, we used global functions to identify all the different testing steps. These global functions were obviously used to avoid duplications. Inside these global functions we started programming / automating the required steps to operate the UI. We used keywords like Click, WaitFor and Enter data for UI elements identified by images. We also made sure that all the images, used during the test-execution, were captured in a default resolution and other related image settings. That required all workstations to have the same display settings to avoid image duplications.
The pros and cons
After the project we did a retrospective to understand what pros and cons we witnessed using ICTestAutomation. You can find the outcome down below.
Pros
- Get started quickly
- Easy to test business processes, not just one application
- Passing parameters makes re-use possible
- No installation on the System Under Test required
- No specific knowledge required for object IDs of controls
- Trendic (Tech) support responds quickly
Cons
- Depends on Windows visual settings
- Not possible to debug
- IDE is perceived as slow
- Not an active community. Maybe there should be a blog post? :-)
- Too boring for real coder?
The learnings
Down below you can find our learnings which we use as tips and tricks for next ICTestAutomation implementations.
- RDP settings for image quality and resolution:
- Use a Remote Desktop-Protocol (or VNC) to setup a fixed resolution
- In RDP settings, make sure you set the performance connection speed to “LAN (10 Mbps or higher) and uncheck all “Allow the following”
- In Windows (SUT), disable “ClearType”
- In Windows (SUT), set “Visual Effects” to “Best Performance” with SystemPropertiesPerformance.exe
- Do not use .ini parameters for images. During development you want to be able to see the images via de IDE, instead of looking up the path manually. - Stable CKEY execution: Use small (milliseconds) delays if you are going to use CKEY (tabs and other keys). Sometimes the SUT is too slow to interpret the keystrokes.
- CI integration: ICTestAutomation can be integrated with external tools. These integrations are realised with the ICTestAutomation module ”ICConnect”. The degree of integration depends on the capabilities of the external solution. When using Jenkins it is possible to start an automatic regression test from Jenkins in ICTestAutomation, after which the results of that test can be received back in Jenkins. To configure this, the Pipeline REST API plugin must first be installed in Jenkins. A curl call to the remote IC Portal system is required to start the execution, passing a callback url according to the asynchronous pipeline principle.
- Version Control: Use a version control system (like git) instead of 1 Windows file share. IC Define regularly crashes when users access the same file(s) during development.
- Email alerting: In the Playback Options of CitraTest you can setup email alerting. Check Alert on Script Fail, Check Enable Alert emails, Fill in Destination addresses and SMTP server.
- Runner execution timeouts: Customise IC Runner execution time out for each client. The ICConsole timeout only applies to the runner that is started via ICPortal. Edit TrendIC.ICConsole.exe.config in folder C:/Program Files(x86)/ICTestAutomation/ICConsole
<add key=”RunnerExecutionTimeOutInSeconds” value=”600" />
Value -1 can be entered for an infinite timeout - IDE IC Define execution timeouts: The IC Define timeout only applies to running test cases using ICDefine. Edit Trendic.ICDefine.exe.config in folder C:/Program Files(x86)/ICTestAutomation/ICDefine
<add key=”TimeOut” value=”600000" />.
Here is the timeout in milliseconds
Value -1 can be entered for an infinite timeout
The result
The end result was impressive. Our client was happy with the result. We built a testsuite of 70+ test cases that ran more than 6 months to monitor the changes of the workstations after upgrades. We found some interesting performance bottlenecks and delivered quality data to give insights on what to do next. It was interesting to have used a test automation tool that is independent of underlying program code, object IDs, or controls. And recognises those objects and controls via OCR and image recognition techniques.
There are definitely some improvements to be made in the developer experience of ICTestAutomation. But I would recommend ICTestAutomation in environments where you have to automate and verify actions over different applications. You will have to change your automation mindset by thinking in images & snapshots instead of object ID’s. Once you get there, you can still apply all your automation principles and practices to make sure you build a maintainable solution.
