Mar

Why UX is Hard

I have been designing interfaces for 23 years, working at various companies throughout my career. In the early years, I would receive wireframes from Project Managers and use my visual design toolset to create aesthetically pleasing mockups. The efficiency of my design work was measured by how quickly I could implement the Project Manager’s requests and whether the mockup I created looked good from their perspective.

As I became more experienced, I began to realize that I could do better than the wireframes I was given. I proposed my own variants with less cluttered layouts, but my ideas were often rejected for unknown reasons. I became frustrated with my inability to change the wireframes and decided to learn more about the Project Manager (PM) process. I studied the PMBOK, obtained the PMP certification, and took on the Project Management role.

What I discovered was shocking to me. Product Managers in my organization were not rewarded for “improving user experience”, but rather for delivering X number of feature in Y amount of time.

Nowhere in the “Scope” document I was expected to include the “Usability” or “User Experience” requirements. In fact these additional requirements once added will delay my project from the timely execution. As a Project Manager, I was advised that if I wished to include them, I needed to ensure that they would not adversely affect the project’s schedule or budget.

In a real world, if you plan to bake bread and receive a requirement to change the percentage of flour, you also need to adjust the amount of water accordingly. Neglecting to make this adjustment would result in an unsuccessful outcome, and nobody would want to buy that bread. However, in the digital world, development teams can do so because the impact of adding a new feature is not immediately clear. It may take months for the impact to become apparent, and for the project management team to recognize and address the issues.

Today, PMs do not create wireframes as often as they did 10 years ago. Instead, they create PRDs, but these PRDs still do not include “Usability” or “User Experience” requirements. Without conducting upfront user testing, it is unclear whether a new business-driven feature will enhance or impair the User Experience. However, as with any other project, it is critical to maintain a balance within the system. If the product team keeps adding features to the UI without considering the User Journey, the User Experience will only deteriorate with each new addition.

Project Managers should be rewarded for delivering value to the business while also improving the User Experience.

Nov

Data Driven Design: Building a Debugger

In early 2021 we released our Playbook Debugger for Palo Alto Networks Cortex XSOAR product and research played a strong role in it.

Due to a data-driven approach and continuous validation process we spent large engineering effort wisely and managed to avoid a lot of pitfalls and save thousands of hours of potential design and engineering rework (which is so costly and time-consuming). Research helped us to maximize user value with the first release.

This project was born when we noticed that our Customer Success team spends a lot of time trying to test Playbooks. We thought that we could make their life easier by releasing Debugger, but we didn’t know whether this feature will make an impact outside of our team. The estimated engineering effort was quite large, and it was not easy for us to commit to such a big product investment without getting more data.

To get more clarity we did the discovery research with our customers.

20+

Design Partners

60+

Interviews Conducted

40+

Beta Participants

1. DISCOVERY

With this Discovery phase we wanted to learn:

What is the actual user need? (do they really want a “Debugger” or just a “Troubleshooting” tool or something completely different?)
How big is this need? (Do our users want this feature?)
Are there segments (groups of customers) who may need this more than others or do not need it at all?

So we started with the research plan that contains open-ended questions, and lets our users communicate their unique experience and their needs. We conducted around 20 interviews, and to our surprise, we discovered that inability to efficiently troubleshoot issues was the biggest Pain Point among our customers (as well among our Customer Success team).

But our Discovery research didn’t end there. While analyzing research reports we noticed that different users want to troubleshoot their playbooks in different product areas. We quickly ran a questionnaire across our users to determine the most important debugging scenario for them.

We found out that there are 2 scenarios where our users may need to debug the code the most:

In the “Incident View” page, when an error occurs in the system, so the data is not displayed as it should.
In the “Playbook Builder”, when a playbook is being built, so the user needs to overwrite context data as a part of playbook building exercise.

Troubleshooting

Playbook Building

Each of these scenarios required a separate design and engineering effort. We communicated out findings to the PM team and collaboratively narrowed down the potential work, so we stayed focused on the features that matter the most. We decided to take different approaches to solve these discovered issuess, and one of them was building Playbook Debugger. We chose a different way to solve the “Troubleshooting” pain point.

2. Product Idea Validation

After research helped us to identify product areas where users wanted Debugger the most, we conducted a series of conversational sessions with our users, asking them how would they imagine “Playbook Building” workflow, and we were surprised to how much congruent and fulfilling were people’s responses:

“I would like to set breakpoints within actual Playbook to see what the inputs are, as well as set Breakpoints within a Sub Playbook.“
“I’m a command line type of guy, I have experience in Perl and Python debuggers, I’d like to see that kind of interface where we can set breakpoints, examine and watch variables, watch registers, change code and alter the flow of things, so we can see behavior of different branches…“
“My expectations would be to set breakpoints, step through the tasks in Sub Playbooks, inspect context along the way.“
Etc.

Together with Product Management team we analyzed these responses and decided on which features we should be designing, and which ones we should put on hold.

3. Design Validation

Our UX team created Debugger design mockups and we validated them to make sure that we are not missing any important details like steps in the flow, interactions, etc.

More discussions we did, more missing pieces we discovered. There were moments when our assumptions were significantly different from what the research results revealed.

For example, there was a “debug” button in the initial design. We imagined “debug” as a separate mode, which appeared to be not what our users wanted. If we released the feature as it was initially designed, we would have run into so many problems:

Reenter same data multiple times every time there is a change.
Need to create/apply/modify transformers each time there is a change.
Need to switch between edit and debug mode constantly.
Unpleasant user experience, requires extra clicks.
Etc.

We didn’t know any of this until we started the validation. And the research helped us learning about the gap before engineering effort was spent.

Design validation raised many functional questions that we haven’t thought about, and the main value of the research at these stage was not to answer these questions, but to get the teams to be aware and aligned on these questions .

Two months after the start of the research, we had much better understanding of a Debugger complexity specific to a SOAR product. We continued to collaborate closely with the PM team on functional areas the debugger should cover.

4. Building Better Flows

Debugger workflows designed by our teams were extremely reach with interactions and co-depencies of data. Debugger elements could behave differently depending on the situation, testing data and conditions applied.

We would validate every piece of our design to make sure that it doesn’t cause any new issues.

The user’s feedback was so valuable, as it helped us to think trough most of the edge cases:

“Are you forcing execution on the left branch or on the right branch?”
“Do I need to re-enter my data in this box”?
“Is this data saved in the context panel?”
“I would not expect the value to be actually override if I save the Playbook.”
“If the output is interpreted by the job as an array, how would you represent that?”

These small details had a major impact on the usability of our product. And the only way to reveal these detail was through doing UX research.

5. Iterative Design Changes

We involved our customers into our design sprints. User Inputs were so precious and eye-opening in validating design assumptions:

“If an error occurs it is usually only occurs on a very specific Incident Type, so I need to be very specific in which data I use for testing.”
“From my experience I’ve noticed that when I try to update mapping data its difficult to get the exact information from the specific case, which is open or assign to me, if there is an ability to choose a case for testing/debugging, that will make my life much easier”
“Specifying an Incident ID for debugging in UI would be much more convenient than exporting and importing JSON”

An image below shows the Evolution of a “Testing Data” panel design.

Iteration1: Product Team had an initial assumption that displaying 3 randomly generated incidents would be sufficient for users to pick one.

Iteration2: Based on user feedback (see above) we saw a need to add a search box, where users can enter the Incident ID or Incident Name.

Iteration3: After we did the demo of the updated search box functionality, users shared with us that it is easy to make a mistake if only numeric value is shown, so they would want to see which Incident Type is it, as well as well as the info, whether this Incident run by this Playbook.

7. Design Elements

We thought that it would be necessary to validate also our high-fidelity design elements.

Despite certain time-proven icon conventions for Debugger products, we decided to learn what is more intuitive for users, and build our designs based on our learnings, rather than just following these conventions blindly.

As an interesting example of an outcome of the user feedback cycle, we replaced “step into/out/over” button bar (left image below) with the contextual navigation “Open sub playbook” link (left image below), see below:

We found out that this link was sufficient and there was no learning curve, compared to the button bar.

8. Prototype Validation

By this time we had an impressive amount of customers who were continuously participating in all rounds of our Debugger research project, so we invited them to try our Debugger Beta and provide their feedback.

Our customers helped us to find the design flaws that we missed in mockups, the users expected a different behavior than we prototyped:

“So when I overrode it, i junked the whole input, I didn’t mean that. I only wanted to get rid of the information in the box.”
“That’s another thing. When I stop it, I don’t want to reset it back to this value. Show me the execution. The green box shall be here.”
“I’m looking for it first, that’s what I was afraid of. A lot of time it displays the previous value. I’m expecting it to show me the executed value.”
“One thing I’m noticing here is that if I want to override a dropdown. I’m not getting same override feature as I have for the label.”

RESULTS

Debugger came out a genuinely successful XSOAR feature. Most of our end users liked the released Debugger feature. Our customers positively mentioned Playbook Debugger in consecutive UX research sessions as their main delight in Cortex XSOAR product.
We saved a lot of engineering effort by validating our assumptions. If we didn’t do research and implemented debugger in a separate mode, we would have ended up with a lot usability problems (see above), that would have been almost impossible to fix locally.
We’ve learned how to do UX research as a team, and created a repeatable research practice that helped doing next research projects faster.

I love the PB debugger. I followed the design process as much as possible. Fantastic job & result! Many thanks for all your work.

Tycho Schmidt

Systems Engineer

Aug

Make User Feedback Visible

User-Centric Design starts and ends with User needs.

Today each enterprise product is a pine tree of teams working together: Product Management, Customer Support, Customer Success, User Experience, Customer Experience, Customer Architects, Sales, Marketing teams (you name it), and each one collects precious assets – feedback.

If you are new to the User Experience field you may think: “Wow, with so many direct customer channels this team should know everything about their Users and their Users’ Pain Points”. But in reality, this can be far from the truth. Smaller product teams have much bigger success in proper capturing (and therefore utilizing) user feedback than large organizations. Problems usually start with scale.

For large product teams more than 80% of user feedback:

is not captured when received verbally,
nor aggregated neither shared across teams,
does not timely reach the decision-making team.

How nice that would be if anyone (PM or UX or developer) could get a summary of a problem on-demand with no effort? Like a magic box, where they can say “Show me all problems that my customers have with dashboards.” And the result will be displayed right away without spinning extra rounds of research or searching across multiple systems. UX-mature Organizations achieve that and stay proactive, while less mature ones remain reactive and are often late with important product decisions.

Step 1: Feedback Collection

Who owns the information owns the world. As a PM how would you fix the problem if you have never heard about it?

So first step towards a UX mature organization would be to create an environment where all teams can capture and pass feedback easily.

Feedback collection and sharing work if:

it is easy to do (requires minimum manual effort: all screenshots, videos, logs are attached automatically, it does not distract people from the main task, etc.)
there is motivation (things get actually improved, customers get rewarded, etc.)
teams are aligned on the goals (engineering for example may not want to receive more bugs, etc.)

If your process requires a dreary annoying manual effort to submit an issue, where the user has to interrupt the task they were doing to collect and download screenshots, videos, logs, precisely describe “ways to reproduce” etc., there would always be a reluctance to submit more feedback from your customers.

Step 2: Create Taxonomy

Let’s imagine that your feedback factory works now and all your team members can easily capture user issues (and motivated to do so!).

Without a universal taxonomy applied to all feedback channels, it would be still very hard to connect the dots. It’s a tedious job to perform a manual search across Git, Jira, Slack, Airtable, Aha, Zendesk, Salesforce, Google Drive, Pendo, Google Analytics, etc. to search and filter for possibly related artifacts just to answer the question you have.

Feature Taxonomy is a good way to ease this process.

The screenshot above is taken from an Aha product and shows an example of Taxonomy applied.

For an average enterprise product I would suggest having 2 levels of a taxonomy:

Epics (Large entities, relatively short list consisting not more than 16-20 items, so anyone can easily browse and memorize it. Usually can be defined ahead of time, doesn’t update often.)
Tags (Detailed entities, related to a smaller system of components, defined ahead of time or as needed, may or may not allow free input, could be applied by machine learning or manually.)

Step 3: Process Change

With UX scaling, product team processes have to change. Hiring a team of UX researchers will not automatically make your product better. Product schedule needs to allow time for discovery, brainstorming, validation. Design teams may create extra work and discard work. Engineering teams should participate in the designing process earlier.

Overall, the team shall agree on the quality bar and be enabled to put extra effort into maintaining this bar.

Image Credit.

Thank you for reading.

Aug

Don’t use generic Personas

When I mentor UX researchers, I sometimes hear that they are not willing to do Persona research just because they’ve done it at the previous workplace or product team, and they already know everything about their users.

What makes your Persona different from any other Persona is the goal of why this persona is created, for example:

Improve Product Experience
Understand users Pain Points and Motivations
Explore product Opportunities
Backlog prioritization
Etc

Even if the goal for your persona was the same at your current and previous projects, the outcome of your research and the data would be and should be different. The same human (that resolves to Persona archetype) will have different Pain Points working with different products, their User Journey will be different, their Goals and Motivations would be different. Most of the Persona data from the other product will be useless, and often misleading.

There are some rare situations where the persona data can be reused. For example, if you collect persona data across multiple products in the organization, the scope of each Persona shall remain consistent and transparent.

In most cases it is recommended to run actual Persona research (not extrapolate from generic provisional assumptions), so your data will be viable and actionable, and help your team to reach their research goals.

Jul

UX Research Done Right

There can be many reasons why some people do not believe in UX research: a lack of understanding of what the researcher does, emphasis on short-term team goals, perception of research process complexity, conflicting stakeholder interests, and prior negative experiences with research.

A key single most important factor to a successful research study is team focus on the Outcomes (and not the Outputs), where people are not afraid to be wrong about their initial assumptions. Unfortunately, as a UX researcher, you can’t really control your stakeholder’s attitude towards research, nor you can change a Confirmation Bias (we tend to ignore evidence to favor information that supports our beliefs).

But there are factors that we can and should control, which will impact the success of a study:

Narrative Bias (and “Leading the Witness”): We tend to rely on storytelling, instead of “storyfinding”, which doesn’t help to reveal our user’s paint points. It’s hard to say “no” to a well-trained elevator pitch that we’ve carefully prepared to convince people of the goodness of our design.
Wrong Persona: Appropriate user group is crucial to get relevant data. It’s not easy to create an environment where you can have access to the right users any time you want, and that’s why a systematic research practice is crucial in an organization.
Wrong Focus: Each research study should have a clear goal, and all individual sessions should be helping to collect data related to this goal. We tend to go too wide with the objective of our study, so the data we get became too complex to digest and understand.
Wrong Methods: There is a reason why a variety of Quantitative and Qualitative studies exist. A method that is appropriate for one project can not be suitable for another one.
Wrong Analysis: We need to be technical enough to understand the message that our technical users provide, or find an efficient way to deliver this message to our technical research stakeholders. Sometimes what we hear is not what our customers say.
Insufficient Sample Size: I do believe that any research is better than no research at all, but an appropriate sample size of the research participants is often necessary to produce a high-quality research result and avoid bias.

Proper research practice provides accurate and reliable data that can inform and improve decision-making, builds trust, provides evidence-based design decisions, and leads to the development of products that are more effective, efficient, and user-friendly. By doing research properly, you can increase the validity and reliability of the findings and make informed decisions that drive the success of your products.

Mar

Validate Requirements Before Design

Classic Product Management Guide PMBOK teaches that:

The quality of work is constrained by the project’s budget, deadlines and scope.
The project manager can trade between constraints.
Changes in one constraint necessitate changes in others to compensate or quality will suffer.

While managing schedule and budget were relatively straightforward, our “scope” was pretty complex. It looked like in the image below.

Image Credit.

Our developers were picking user-stories to implement every 2 weeks during sprint planning. As a PM I always tried to find better ways to deal with that “scope” that contained so many unknowns and dependencies (not to say that it was changing every second week). After years of trial and error, my PMBOK triangle looked like that:

Product Managers produce great functional requirements, but many details can not be foreseen upfront.

Actual User Journey almost never looks as the one we imagine.

Actual research helps to uncover gaps in the:

Domain Knowledge
Requirements
Design
Implementation

Thorough upfront design and validation are more expensive and time consuming than the traditional process of doing design-development cycles. Many design deliverables need to be discarded and it can be seen as a “waste” by some people. But it’s not a waste, it is a saved engineering effort on the big picture, if the final goal is to release the product that users like and want.

Image Credit.

Some product teams prefer it to do the other way around, which results in much greater costs, as it not only involves design change, but also change of the code and QA, change of design system patters, change of documents, not to mention the experience that end-users get.

User-centric approach costs more and takes longer in the initial phase, but it is very rewarding later.

Feb

How UX mature is your Organization?

15 years ago I worked as a UI design lead, transitioning to a PM role. My team’s Scrum Backlog looked like the image below:

Image Credit.

And the product delivery process in my team was:

Business Analyst / PM creates / updates a Backlog.
Every 2 weeks engineering team chooses which stories they want to code.
Engineering team starts coding the user stories.
Engineering team may ask a designer to help, but most of the time they will code these stories on their own.

As a PM with a design background, I suffered seeing the outcomes. Usability and design quality was untenable, and my very first step to increasing the quality was to incorporate design effort before the engineering time was spent, so designers become a part of the entire scrum cycle (before, during, and after the sprint).

As we managed to produce nicely designed pages, my next step was to start collecting actual feedback from the end users. When I got my very first user interviews, in spite that the UI looked good, I’ve discovered that nobody wanted to use our product, their company purchased it globally and they had no choice. I went back to my manager and shared the results of user testing, hoping that we could change things so our users don’t suffer, but I was told that this is not in the company’s interest to invest any additional effort in increasing user satisfaction and that we are also providing personalized training for $1500 per person.

That day I’ve realized that UX is much more connected to the Product Strategy than to UI or any other discipline.

UX Shall be included

To create great products, UX shall be included in all phases thoroughly, and not just be a layer between PM and Dev.

Organizations often try to “save time” and don’t validate their requirements and designs well. The desire to ship something sooner is understandable, but cutting corners will inevitably show up in the final product.

Just having a decent UX budget and high skilled team of researchers and designers doesn’t automatically makes your product great. What makes the difference is how effectively this knowledge is used across other key teams.

UX vs PMX (c)

I’ve noticed that a lot of organizations do “PMX” (c) rather than “UX”. This happens when PM doesn’t use User Research data in the design process. UX designers can perform their high-quality design work, but this work will cater just PM assumptions, which can (or can not) match user needs.

Questions that a UX mature organization will ask itself:

Does your Organization collect enough / good enough data / metrics to inform design / strategy decisions? Is this information easily accessible to the team members?
Does your team know how happy your users are with every key feature at any moment (proactively collecting and structuring the UX insights)?
Does your team validate the requirements?
Does your PM team consider improving the actual User Journey (along with catering standalone business requirements)?
Does your design actually solve user problems? Does it bring any new usability issues (how well it is validated)?
Has your dev team implemented the design the way it was designed?
Does your roadmap have the bandwidth to take care of a technical / usability debt?

Low Maturity Signals

Nielsen recently released a video with few other signals of Low UX maturity in the organization:

You must repeatedly explain the difference between visual design and interaction design.
Usability testing is done late, and only on working designs (never on prototypes).
You never do field studies.
UX practitioners are excluded from product meetings.
UX Processes and research are neither viable nor automatic.

There are many more variables to a good UX rather than having a good UX team.

And if you disagree with me, this little video is for you.

Jun

Putting Personas to Work

When I shifted my career from UI to UX design, the very first thing I’ve learned about was Persona. And I immediately liked the idea of having one, because in the past I saw products fail due to misalignment with customer needs, I saw product teams do not agree on what a customer “need” is, and Persona seemed to be a great tool to prevent this. It is beneficial to think of personas and present them as an alignment tool rather than a research deliverable.

I’ve read all possible books about building the Persona, but what was missing for me is practical information about what I should do to make it work. When I searched for “Persona Template” in Google, it gave me a set of very similar results, so I thought this is something I should start with and I tried to fit my research into these templates.

So I took one of these templates and worked hard to construct my Personas this way. I didn’t have a lot of success with trying to adopt these in my teams and as a designer I was demotivated with using Personas and then abandoned Personas for years, experimenting with other frameworks.

Sometimes Personas Are Too Personal

Later I came back to designing Personas and one thing that I’ve realized that “impersonating” results of my research doesn’t help in any way and could be misleading. I work in the Enterprise IT space and if I choose a photo/image for each of my Personas the whole set will look like that (see below). Using fictional images and names without a purpose forces other team members to memorize them and this on its own creates a barrier in adopting Persona.

Image Credit: Airbus

Alan Cooper in his book “About Face” says that “Personas are segmented along ranges of usage behavior, not demographics or buying behavior”.

For the consumer products however using visuals when constructing personas makes more sense, and not just because of the demographic (which can be a reasonable differentiator on its own), but also because of the environment where the products are used in.

Image Credit: Plantronics and BBtalkin.

What Makes a Good Persona

Personas are not a one-size-fits-all tool. They should be used with a specific, well-defined goal in mind. For Personas to be useful, the data captured in a Persona should reflect the goal for that Persona and the scope of work it is meant to impact. (NNgroup)

The goals can be:

Understanding who your users are,
Increasing awareness of Personas in your organization,
Team alignment on common goals,
Discovery of segments of customers which needs / pains are unmet,
Backlog prioritization,
User base segmentation,
Etc.

Persona creation process takes time and should be based on the actual research data and not on what we think we know about similar archetypes/jobs. Persona will change over time if product keeps changing and building one is not a one-time effort, but rather a continuous process.

Jobs-To-Be-Done

Using “Jobs-To-Be-Done” helped me to make my work research functional, and my research outcomes finally were adopted by the teams. Fantastic video explaining the value of Jobs-To-Be-Done by Clay Christensen is below.

An example of a Jobs-To-Be-Done template is below.

Image Credit Tony Ulwick

What was always missing for me in the Jobs-To-Be-Done framework, that it’s not focused around the actual user’s goals and pains, so product thinking becomes very functional, but not user-centric.

My Persona Template

I found it helpful to adopt some practices from “Jobs-To-Be-Done” framework when I build Personas. So I use the following principles. Personas should be:

Concise
Based on the facts
Include information that helps to drive design/product decisions
Functional
Presenting user Goals and Pain Points

A good example of product Personas is provided below.

Image Credit Enisa.

So I’ve created a Persona template (and it is connected to the actual research artifacts in my Airtable).

So the template above contains of:

Persona. I like to use “Role” instead of a fictional person name, because it reduces cognitive load and makes the document transparent.
Journey / Sub Product. Same persona can be primary in Journey A and secondary in Journey B.
Primary / Secondary. The number of primary personas should be as little as possible.
Description. Facts that help to describe a certain archetype.
Contributions. Is taken from the “Jobs-To-Be-Done” framework and helps to describe their activity with a product.
Goals / Needs. Actual Goals that your users share or are observed by researcher.
Challenges / Pain Points. Actual user pain points that your users share or are observed by researcher.
People. This connects straight to my Airtable bases and lists all interviewees who belong to this Persona type.
Notes. Any other related information (user’s actual job titles for example).

Please let me know in comments what do you think about my Persona template and what Persona goals do you set in your organization.

Jan

Why Do UX Research

Can you imagine a world where AI creates software in seconds by reading the person’s mind? There would not be any costs in achieving perfect design, as your project could be modified as many times as you want in seconds. There would not be any middlemen, who may have a different vision of what you are trying to achieve. There would not be other users with different needs that this product must cater to.

One day we may see that, but today there are significant costs associated with creating software products. There are hundreds of decisions made each week within the product team, and each of these decisions impacts your user’s product experience. Today many products fail despite the best design efforts which can fail as well.

The needs of customers and what a product team wants to build can sometimes differ due to several reasons:

Limited Understanding: The product owners may not have a complete understanding of the target users’ needs and preferences, leading to a mismatch between what is built and what is needed.
Different Perspectives: The product team may have a different perspective on how the product should work or be used, leading to differences with the target users’ expectations.
Technical Limitations: Technical limitations may prevent the team from building a product that meets the full range of user needs.
Misaligned Priorities: The product stakeholders may have different priorities or goals than the target users, leading to differences in what is considered important.
Market Trends: Market trends and new technologies may shift the focus of the product team, leading to a product that does not fully meet the needs of the target users.

Having viable UX research practice in your team will not guarantee that the product will be successful, but it will significantly decrease the chances of uncertainty by providing the user feedback insights ahead of time, before significant engineering effort is spent.

Conducting UX research will help to gain a deep understanding of target users and their needs, which allows product team to build products that not only meet the technical requirements but also meet the needs of the target users.