What Are Actions?
Actions are the atomic operations of automation — clicking a button, typing text, finding an image, waiting for an element to vanish. Where states and transitions describe the structure of an application, actions are what the engine actually does when traversing a transition.
Key Concept: Actions Target Elements Visually
An action does not point at a fixed coordinate or a brittle selector. It targets an element by its visual appearance — the engine first finds the element on screen with template matching, then acts at the matched location. This keeps actions robust as the layout shifts.
Categories of Action
Conceptually, actions fall into a few families. Together they let a transition observe the screen, manipulate input, branch on conditions, and orchestrate other work.
Find
Locate an element on screen, or wait for one to appear or disappear. Find is the foundation other actions build on — it resolves the target before any interaction happens.
Mouse & Keyboard
Drive real input: click, move, drag, scroll, type text, and press key combinations. These produce the actual interactions a user would perform.
Control Flow & Data
Branch on conditions, loop over items, and read or write variables — so a transition can express logic, not just a flat list of clicks.
State & Code
Coordinate the model itself (navigate to a state, run another workflow) and reach beyond the GUI when needed (run a script, invoke an AI prompt).
The Find-Then-Act Pattern
Most interactions follow the same shape: resolve where the target is, then act there. Understanding this pattern explains why visual automation tolerates layout changes that break coordinate-based scripts.
- 1.The action names a target — typically a state image
- 2.The engine finds that target on the current screen
- 3.The action executes at the matched location (e.g. clicks its center)
- 4.Because the location is resolved at run time, the element can move without breaking the action
Self-Healing
When an element cannot be found at the expected confidence, the engine does not have to fail outright. Qontinui includes an optional self-healing system that attempts recovery before giving up.
Action caching
Successful element locations are remembered, so a repeat lookup can replay instantly instead of searching from scratch.
Visual search fallback
When exact matching fails, the engine retries at lower similarity thresholds and multiple scales to find a near match.
LLM assistance (optional)
A vision model — local or cloud — can locate an element from a natural-language description when template matching alone is not enough.
Note: This page covers the concepts. For the complete catalog of action types you can drop into a workflow, see the Action Types reference.