Appium Tools for Rapid Development of Functional & Performance Tests.pdf

Appium actually lets you retrieve some raw performance data just using the regular old Appium APIs, and I recommend checking that out.

1. Appium Tools for Rapid Development of
Functional & Performance Tests

So, I just want to give a little bit of a refresher on the types of testing that we’re considering
today. First of all, we have functional testing, which I’m sure many of you are quite familiar with
already. But if you are kind of wondering what’s underneath this term of functional testing, we
can go back to the word function in its mathematical sense. So, if you’ve ever, you know, done
algebra or calculus you’ve seen, you know, formulas or equations that look like this, you know, f
(x) equals something else (equals why in this case). In this schema, f is the name of a function
and functions take an input, or one or more inputs, but from a mathematical perspective usually
one input, and then you can compose multiple functions together to deal with multiple inputs.
Anyway, you’ve got your input and then you’ve got your results. So, you know different functions
are defined by the different ways that inputs lead to outputs.
So, from the perspective of functional testing, we might re-categorize this as saying functional
tasks is asking, you know, does f (x) actually equal y? So, when I take my application, which we
can consider to be a function or a set of functions, and when we put in an input of a certain kind,
whether that’s a tap or a whole series of actions, is the correct output produced?
So, this is what we mean by functional testing. It’s basically, given the right inputs, does my
application behave correctly? And, sometimes given the wrong inputs, does my application

2. behave correctly dealing with those wrong inputs. Are the error conditions handled appropriately
making sure the app doesn’t crash or something like that.
There are a bunch of aspects of testing which people don’t usually include when they’re thinking
about functional testing.
One of these is time. You know, before we asked does f (x) equal y. Well, we didn’t ask “and
how long did it take to equal y?” You know, this is something which is important for users but not
really an aspect of functional testing if it takes 10 seconds or 20 seconds. The functionality still
worked.
Another aspect of an application that doesn’t usually show up in functional test is appearance.
Does an app look good is it usable from a visual perspective? Does it contain any visual
regressions – things like that? As long as the functionality of the app isn’t impacted, usually
appearance isn’t considered as part of functional testing. Security is another issue. Security and
functionality go hand-in-hand obviously. Security could be viewed as an element of functionality,
but very often it’s not and so very often, for security testing you get to a different layer of the test
stack and consider it as a separate thing that is a responsibility of another team than the people
writing functional test. Things like that. So, there are these aspects that aren’t often considered
in functional testing.
This is often reflected in the usage of Appium, because Appium is fundamentally just a UI
automation tool. It’s not a test framework. It doesn’t give you all the features that you need for
writing and running tests. Its primary focus is to let you automate the use of a user interface.
So, in other words, it’s kind of like in the mathematical formula we had earlier. We had this
function, which you could think of as your application, and we had this way of passing in an
input which we called X and then determining an output which we call Y. And Appium is just a
way of letting you input X into your application and letting you figure out what Y is, what
happened on your application after we input X. So, in other words, it kind of provides the I/O, or
the input and output, for functional testing. That’s why Appium is, I would say, primarily used for
functional testing, even though it can also be used in other kinds of testing as well. One of the most important other kinds of testing is what we call performance testing. So, if we
go back to our little schema here, which we were using for functional testing, f(x) equals Y, we
could ask additional questions beyond does f(x) equal Y and that gets us into the domain of
performance testing. So, these are some of the questions we might ask within the domain of
performance testing.
Okay, so f(x) was Y. We determine that with our functional testing, but did it become Y fast
enough? Are we going to meet the users’ expectations of speed and responsivity? Did f(x)
maybe consume too much memory? Now this is something that the user might not see
themselves, but might impact the operation of the system, might impact the further operation of
the app, or might cause a crash down the road. This is something internal to the operation of the
function, not something that is visible within the UI, at least not until it gets to a crisis point.

3. On Android, you might have an out-of-memory killer that shuts the app down. Things like that
might happen. We might ask: what are the various kinds of bottlenecks inside the operation of
the application? We can look at the application from an internal perspective and ask, you know,
when I tap this button and when I waited for this result to appear on the screen, what were all
the parts of my application internals that that were navigated through that were called, and how
much time did the app spend in each of those? We can figure out bottlenecks on our
application as a way of trying to point out problems or give us areas to potentially work
on for improvement of the performance.
We could ask beyond the app itself, did running our application have any consequences on the
device at large or the system at large? What could these negative consequences be? Could we
be using way too much CPU, such that other applications are negatively impacted or that the
battery is drained too quickly? Things like that.
We might also ask about how much data our app consumed, because if we’re developing our
apps with the assumption that all of our users are on Wi-Fi speeds, we might run into an issue
where our app is downloading so much data over the network that it becomes unusable in
cellular conditions, 3G conditions, or 2G conditions.
So, this would show up to the user as something like slowness, but asking how much data
was drawn down over a period of time is a really good way at figuring out again where we
could optimize or where we’re making assumptions about our users’ experiences. So,
these are the kinds of things that we talked about in performance testing.
Now since we’re talking about HeadSpin today in this webinar and it’s being hosted by my
friends there, I took a screenshot of the HeadSpin application.
This is a UI, which is given to you after you run one of your Appium tests on HeadSpin. I know
that the slide is probably showing you a bunch of stuff that’s too small to read and it’s not
important for us to go through this in detail. My point is that all of these little metrics and graphs
and potential issues that you see here are the kinds of things that you care about more in
performance testing than functional testing.
You could have an app work totally fine on a functional level but two runs of the same
application and two different conditions might give you a very different set of
performance data here. In some situations, it might be really concerning and some situations
might not be.
For example, in this example, HeadSpin has automatically found some issues that we might
want to take a look at. So, it’s evaluated various aspects of the performance data of the
application that we didn’t have to capture ourselves. It was kind of capturing them automatically
as we are running our tests, but it’s sharing that this app – it’s the Walmart app in fact – had a
loading animation on the screen for more than a second, which is known to be a potential spot
of frustration for users. So, it’s saying we’ve got 11 counts of this error, so hopefully the Walmart

4. team is taking a look at this kind of thing. I know from experience, having played around trying
to automate this app, that it is painfully slow.
We can look at a lot of different issues not just as a bear warning but over time as well. This is
the kind of stuff that we’re talking about when we talk about performance. Most of these bits of
data pertain to network data and how long it took things to download and how long it took things
to display on the screen.
We could also dig even more deep into the application internals. Here’s a bunch of timelines
and some graphs of internal system resource usage. So, we’ve got a graph of the CPU usage
over time.
We could see for example, and of course if we were actually on the website, we could drag the
mouse over here and be greeted with a screenshot of the application at this particular time, so if
we saw a particular spike in CPU, we could put our mouse over it and say, what’s happening
here within the application? There are other ways to do this looking at process graphs and
things like that too that are pretty interesting, especially when you’re digging into the nitty-gritty
of the performance of your app. We can look at memory as well as CPU. Things like that. IO. So, there’s all kinds of pretty standard types of performance data that is really interesting to look
at once you kind of dig a little deeper than the functional level.
Appium actually lets you retrieve some raw performance data just using the regular old Appium
APIs, and I recommend checking that out. Some of the early editions of Appium Pro were on
this topic. There’s an article on Android and an article on iOS. If you just search from
appiumpro.com you can go and find them or I’m sure Google will take you there directly as well.
So, check that out.
As I showed you in the screenshots, one of the cool things about HeadSpin is that they actually
capture all this performance data automatically while you’re just running your functional tests, so
you don’t have to do anything to capture their performance data. That’s all done kind of under
the hood transparently to you. So, it’s kind of nice because you can focus on writing your
functional test asking does f(x) equal y, but then in addition to that just by running on HeadSpin
you get a report back that says by the way, here’s how long it took for y to show up on screen
and here’s all your bottlenecks and here’s all these potential issues. So that’s a really great thing about their service that I really appreciate. Something else to
mention is that I mean this is true for all kinds of testing, but I think it’s especially relevant for
performance testing. You can run one-off performance tests, but it’s often more useful to think of
performance as something that shouldn’t just be tested at discrete points at time but should be
monitored and should be automatically checked. You know, you want to make sure that there is
nothing maybe in the back-end of your application that’s changing that is causing performance
issues. Things like that. So that’s just a brief overview of functional performance testing. I thought it’s interesting to dig a
little deeper into that because I know that we can often throw the words around but may not

5. know the concepts super clearly. Without further ado, let’s move on to some examples of some
of the tools that I use to develop Appium tests and to make sure that they perform well.
Appium Desktop
So first thing I want to talk about is Appium Desktop. Appium Desktop is a user interface for
Appium that you can use without even needing to download anything from the command line or
open up your terminal. It’s a totally self-contained application.
So, if you’ve never used Appium Desktop, check it out. It’s at
github.com/appium/appium-desktop and you can download binary releases for Mac, Windows,
and Linux there. Rather than show you a bunch of slides, I’ll just open it up and do a little demo
of some of the things that I use it for when I’m developing Appium tests.
So, I apologize – things probably will show up kind of small on my screen. I’ll maximize things as
much as I can. Actually, I wonder if I can. I can’t change the zoom within the application here,
because it’s not actually a web browser. So, this is Appium Desktop. When you open it up, it
greets you with this kind of screen that says hey, I’m Appium what host support do you want to
run me on. I’m just going to go ahead and close down some other windows too so that we can
see it in all its beauty. Here we are. You can see my very dirty desktop as well. So, here’s Appium. What we can do
once we’ve got it downloaded is we can actually start an Appium server. So, an Appium server
is running. It says welcome to Appium version 1.15.1. So, I don’t need to run Appium from the
command line if I don’t want to, and I could run an Appium test from my IDE or anything else. It
says its running on Port 4723, which is what I selected, but usually what I use acting desktop for
is not just as the server, but I use it to inspect my application so that I know how to find the
elements in the most efficient way possible. To do that I can go over here and start an inspector
session.
Once this loads up, I basically have a choice of server here. I can choose an automatic server,
which is the one that’s already running. I can choose a custom server, and I can even select
cloud providers to run on including HeadSpin. Then I basically set up an inspector session by
typing in the desired capabilities, or I can use a saved capability.
I’m going to choose a set of capabilities that I implemented for this webinar so we can see we’re
going to run on iOS 11.4 using an iPhone 8 simulator and I’ve got this application that I
developed that I’m testing, and I’ve got this other capability here which I’ll explain a little bit later
on. Once I’ve got my capability set, I can go ahead and start session.
I’ll go ahead and throw up the iOS simulator so you can see what stuff is happening. Obviously
the Appium logs are scrolling by in the background.
So once the session loads what we have is a screenshot of the app here and I can hover over
all the different elements, just kind of like Chrome or Safari or Firefox devtools. I can tap on one

6. of these elements that Appium knows about and I can see a whole bunch of interesting
metadata about it like its name, its label, its position.
But the most interesting thing is that Appium Desktop actually gives me suggestions of how to
find this element in my test. So for this element, I can actually find it by accessibility ID, which is
preferred and here’s the selector that I can use to find that.
So, I can actually test this out if I want just by tapping on the element here so I can click tap and
we can see that we got to a new screen, new elements, new UI tree here, and the tap was
actually successful. So let’s try and find something else.
Here’s a save button. Now, it does have an accessibility ID, but as you can see in the XML tree,
it’s accessibility ID is not unique, because it’s the same as the element that’s its parent, and
that’s because I’m using react native to develop my application. So, it just kind of does this –
whether I like it or not.
So, the selector that Appium has recommended in this case is X path, but it hasn’t given me a
kind of unintelligent XPath selector. It’s given me one that has some unique information in it
namely. Here’s the name of the element. The name itself isn’t unique, but the fact that it’s the
second one of those on the screen makes it unique.
Now, some feature that I use that I know a lot of people don’t really know about is something
called the search for element dialogue. So let’s say I didn’t really know that this XPath was the
suggested one or I was trying one out in my test and my code, but I’m getting an “element not
found” error. So I can load the app up in this screen. I can click search for element. I can choose
my locator strategy.
So, let’s choose accessibility ID, and I’m going to go ahead and type in what I thought maybe
was the name of that particular button. Then I can click search, and Appium will come back with
all the elements that it thinks correspond to this element. So actually, there’s two – so they sort
of occur in more or less the same space on the screen. So, the fact that there’s two elements
here could indicate to me the cause of a bug in my test script.
Maybe I assumed there was only one or maybe I was trying to interact with one of them, but I
was actually interacting with the other one. Of course, with this selected, I can also tap it or do
different things here. So that’s kind of a fun little tool within Appium Desktop.
Something else you can do within Appium Desktop that some people don’t know about – this is
with more recent versions – is we actually have an actions tab over here.
This is when we want to perform Appium commands that aren’t really related to elements per
se. So Appium has all these different actions that do with the device or the session. So, let’s just
pick session here and you can see we have all these different groups of commands. What
should we try?

7. Let’s try orientation. Let’s get the orientation of the app. So, we get the result commanders
return result portrait. Okay, that’s cool. I wonder if we can even set the orientation. Let’s try
setting it to be landscape and voila. So, I’m going to change it back to portrait now.
So, you can do a lot of things with Appium beyond just find and interact with element
specifically. So have a look at this actions tab. There’s a bunch of interesting stuff in there.
Another kind of fun tool, that’s especially useful when you’re developing tests is the recorder.
So, we got something here already. I’m going to clear out. It’s actually saved my last action. But
if I turn on the recorder, I can pause it or start it again, and if I perform some actions like for
example “go back”.
We can see that that command has been added here in text, but this is the command in a
particular language so I could if I wanted kind of copy and paste this into my test code. This is
python. We could use Java or anything else, and if you want to take this and actually kind of
make a full test out of it, that you could copy into your ID and execute. You can show and hide
boilerplate code.
So, now I’ve got all of this in the context of the correct capabilities that I use to start the test and
actually even puts it into a nice test method for me. Here’s the back command that we used.
Let’s actually go to the web view page here. So, I'll tap this.
You can see that the element was found and clicked in the code as well. So that’s a pretty useful
feature of Appium Desktop. Currently, there is no ability to play this back within Appium
Desktop, but that’s something that we’ve been working on and we’ll release at some point.
So, one of the things that – I’m going to go ahead and close out the recorder here – one of the
things you’ll notice that if you’ve got a Webview that basically it just shows up as this view
element here. It doesn’t give you a bunch of context underneath it even though there might be
elements within this.
So, Appium Desktop doesn’t actually support inspecting within a Webview, at least any
elements that aren’t automatically promoted to native elements like this one here. Apple is
promoted to a native element even though it’s actually a web element. But, we can’t use the
contacts API within Appium Desktop, and the reason is because you can actually already do
that really easily using Safari for iOS and Chrome for Android. So just as a way of highlighting
that let’s go ahead and open up Safari. I could make it a little smaller. So, here’s Safari. If I go up to develop, I can see that there’s
actually an iOS simulator in this list, and there are two web contexts within it about:blank and JS
context. Now, I’m not exactly sure which one of these is this, so I’m going to first of all type into
the box and go to AppiumPro.com.
Sadly, with the most recent version of that code, there’s a bit of a delay in sending keystrokes,
and this is a bug and XCUITest which hopefully Apple will be fixing soon. Maybe I’ll just type it in
manually instead of waiting here.

8. Oh, it looks like the whole simulators is occupied. Well, that’s not very happy is it. Now it’s
actually making my system slow. Okay. Well. We’ll go ahead and just pick one of these web
views, and we’ll see which one is the right one.
So, I’m going to pick about:blank, and now I get actually a web inspector here that shows me
everything that’s available within sight within the Webview. So, this one doesn’t actually have
any elements in it. So, potentially this was not the one that I wanted to work with. Let’s try the
other one.
This is just the JavaScript context for that one as well. So, I think that probably because the
simulator is hanging, it’s not actually showing me the element within this Webview.
Anyway, this is how you would do it. I apologize on behalf of Apple for their untested software.
So yeah, you can do the same thing with Chrome on Android as well. I actually have some
Appium Pro articles on how to do that. So, this was all basically how to inspect your
applications, how to exercise them using a visual interface, and you can do both the native
aspect your application and the web aspect using Safari Chrome. So, we’ll come back to this. I
might have to quit the simulator. See if that works. Okay, we will try that again later. So, back to the presentation here. Something else that I want to talk about this morning is a new
tool that has been released recently called Web2driver. So, Web2driver is a new kind of Appium
client. So, you’re thinking Appium client, you’re thinking okay, there’s the java client, the python
client, .NET, ruby, you know couple JavaScript clients. These all run from your test scripts, which
you run on the command line, right? Or from your editor like IntelliJ or whatever.
Web2Driver
So, this is an assumption which has been true for a long time – until now. So, Web2driver is
actually an Appium client that runs within a web page. It’s located at HeadSpin’s GitHub
organization which is called “projectxyzio/web2driver” on GitHub. So, you can check it out.
Here’s a little image of the readme.
As it says, Web2driver is a pure JavaScript web driver client that can run in a browser. One of
the interesting facts about Web2driver is that because it runs in a browser, it kind of depends on
the browser’s cross-site request to speak to the Appium server. But, because of browser
standards to do with security and making sure the bad people can’t do things you don’t want,
your browser can’t just use JavaScript to talk to a different server than the one that served it the
webpage to begin with. So if you want Web2driver to be able to speak to your Appium server, you actually have to start
the Appium server with a special flag, which is called allow CORS, which is the cross-origin
request security. I don’t remember what ‘S’ stands for – something about cross-origin requests, I
think.

9. So, Appium doesn’t have this turned on by default, because it is again potentially a security risk.
You could go to a web page, and somebody could start running tests on your Appium server –
running on your machine without your knowledge. That’s pretty scary. So, you have to start the
Appium server with that special flag to make sure that that’s able to happen. Otherwise, you can
bundle this web driver package into your own JavaScript projects that you can run within a web
page. So, that’s pretty amazing. The Web2driver code looks very much like something you’d expect from WebDriver IO, and in
fact, we built Web2driver on top of the same code that WebDriver IO is built off of. So, it uses all
that logic, which has obviously been quite battle-tested over the years. So, running a test looks
like defining your capabilities, starting a session, finding elements. Everything is promise-based,
so you can use async/await to keep your code clean.
So, it’s pretty much a standard WebDriver and Appium client. I say Appium client, but really
could use this for Selenium too. Although Selenium doesn’t support the allow CORS headers.
So, you’re kind of out of luck, unless you put a proxy in front of Selenium that you built.
Why do we want a web-based Appium client?
Well, what if we could build Appium tools that require nothing but a browser – no downloads, no
command lines – just your browser. This is, I think, pretty interesting.
So, this is exactly what we’ve been working on at HeadSpin. Here’s a little video of something
called the HeadSpin Recorder, which is I guess in beta now. So, if you’re a HeadSpin user –
talk to your account people and see if you can get access to this soon. But, the idea is that you
load your app, you define the type of device you want to use, and then this recorder interface
pops up.
The interface looks different, but it’s actually very similar to Appium Desktop, in the sense that it
gives you a kind of point-and-click interface to use with your application. But, what’s happening
is that this is all just taking place in your browser.
So, you just go to your HeadSpin account, go to your recorder tab, and you can load it up. You
don’t have to download Appium Desktop or anything else. And, what’s really interesting about it
is that not only does the recorder record your actions, but then you can have them play back
and you can even set them to play back at set intervals. So, you can essentially define your own
Appium-based test without using any code. So obviously there are limitations to any kind of
no-code approach, but this can get you pretty far and it’s been pretty stable so far as I’ve been
playing with it. So, this is just kind of showing you some examples of you know, clicking on things and finding
an element – very similar to Appium Desktop. You can see the actions proceeding here. So,
there’s all kinds of different interactions you can use. You can even do swipes. So, here’s a
swipe happening, and then you can even find elements by image. This is one of the cool things
about the HeadSpin Recorder. You can select a region of the screen that may or may not be an

10. element, and then that screen reading will be passed to Appium as an image, and it can be
tapped during later runs.
So, in the video, the tests are just kind of finished and said, you know, I’m done, and now the
recorder is automatically moving into the review phase where it’s loading up the device again
with a new session. It’s going to replay all of these steps for you to kind of see – oh, yeah, this
works like I expect. I’m going to sign off on this and now can schedule the test.
So, it’s going to skip ahead, you know, so here we are walking through the different steps for the
application. We’re in the middle of a swipe here. Then, again once we’re done, we can create a
report and decide how frequently we’d like this report to be regenerated by re-running all these
commands.
So again, this is kind of an example of the kind of thing you could build with Web2driver
because it’s a full-fledged Appium client. So, you can build these great experiences on top of
Appium just in a browser that make it much more accessible, require no special software, are
accessible from Mac and Linux and Windows and any operating system that can run Firefox.
That kind of thing. So that is the recorder.
Last thing I want to talk about is something called Event Timings. Event Timings is an Appium
feature that can be helpful for digging a bit deeper into the performance of your test itself, kind
of giving you a little idea of what’s going on under the hood of the test- the different steps that
Appium is taking that you may not even know it’s taking – that can help you discover where
there are potential bottlenecks within your test code.
Events API
So, the fundamental idea here is that Appium takes time to do stuff.
This is pretty obvious. Not exactly mind-blowing, but a question we can ask is well, how long?
How long does it take to do those different things? We’re just sitting there running a test, and it
looks like it’s taking forever. Obviously, something’s happening. How many things are
happening, and when are they happening? And what is the thing that is causing the holdup?
So, the Appium team has developed something called the Events API to help us determine this.
Now, when you start a normal Appium test, you use capabilities like this. This is the kind of stuff
that we’ve been using today.
To turn on the Events API, basically you add a new capability called “eventTimings” and just set
it to true. Now, when “eventTimings” is set to true, then you have the ability to get a bunch of
interesting timing data from Appium just by calling the “getSessionDetails” command. So, this is
I think in the Java client it’s “driver.getSessionDetails” – something like that.
Now that response will have a bunch of other information like your capabilities and things like
that, but there will be this key in the response called “events”. Within the events object here, we
have basically two main parts.

11. One is a list of commands. So, every command that you send during the course of your test will
get added here, including the command name and the start time and the end time, which you
could obviously use to figure out how long did each command take. And then there are a bunch
of other kind of special events that we’ve built into the Appium startup flow.
For example, this is an iOS test that I got this data from. So, there’s an event in the startup of an
iOS test called ‘xcodeDetailsRetrieved’. So, we log that when you have the ‘eventTimings” API
turned on, so that you can kind of see when did this happen, and then you can see the next
thing that happened, and so on. You can see that the last thing that happened within this
particular test that I ran was that I set the orientation, or the orientation was set automatically, I
guess, by the Appium server when I start it up. So, we could plot all of these on a map or on a timeline and figure out: where are the spaces in
between these different events? But you don’t actually have to do that manually, because we
have something called the Appium Event Parser that will generate a nice timeline for you from
that data. So, you can take that data, use it on your own, whatever you want to do with it, but we
can also generate a nice timeline.
So, here’s an example of the output for the Event Parser for some of this event data that I
pasted into a file and ran through the Event Parser. So, you can see that all of these different
events happened – xcodeDetails were retrieved, the app was configured, the reset was started,
the reset was complete. Then we actually see this is all to scale. We can see there’s a gap here
of almost three seconds – between resetComplete and logCaptureStarted.
So, what was going on between here? Did it actually take three seconds for the log to start or
maybe the Appium team needs to add a little bit more granularity in here so we can see what’s
happening? Or maybe simultaneously the simulator was starting? Maybe that’s what took some
time.
Maybe we’ve got other gap of almost five seconds for the app to be installed and for us to start
launching WebDriverAgent? Then by far the biggest amount of time taken here was the
WebDriverAgent start. So, between start attempted and session started, it was almost seven, or
a little over six, seconds.
So, this is pretty interesting. Let’s run a quick demo of this actually. So, to run this demo, I’m
going to go ahead and start up a test again, and I’ve got the eventTimings capability turned on
here.
Article Source:

This article was originally published on:
https://www.headspin.io/blog/appium-tools-for-rapid-development-of-functional-performance-test
s