Effects of Developers’ Training on User-Developer Interactions in Information Systems Development

Effects of Developers’ Training on User-Developer
Interactions in Information Systems Development

Jennifer McCauley, Sharoda A. Paul, Andrey Soares
April 20, 2006

IST541 – Qualitative Research in IST
College of Information Sciences and Technology
The Pennsylvania State University
{jmccauley, spaul, asoares}@ist.psu.edu

Abstract: The importance of user-developer interactions during the development of an

information system has been a long-running theme in information systems research. This

research seeks to highlight a gap in the current literature: the contribution of the developer’s

formal educational background to the relationship between developers and users. Using an

interpretivist epistemology, the researchers employed qualitative interviews to examine how far

developers’ perception of the importance of interacting with the user was influenced by their

formal education, or the lack thereof. Interviewing both formally and informally trained

developers, eleven categories of interest were identified as pertinent to determining the

developers’ beliefs about the importance of user interaction. Three of these categories were

explored as promising for future research: academic background, work experience, and

developer’s access to user knowledge. This research has implications for education of

information systems developers as well as for industry interested in hiring software developers.

Keywords: information systems development, user interaction, developers training

1 INTRODUCTION

User participation and involvement has been a long running theme in information systems (IS)

1

research, specifically with respect to the development of information systems. Research on the

correlation between user participation/involvement and success of information systems

development (ISD) has produced contradictory results. While some researchers have found that

greater user participation leads to more user satisfaction, there has been no agreement on whether

greater user participation guarantees successful ISD.

In examining the effect of user participation, much attention has been paid to the relationship

between users and developers, especially on the subject of how they share and transfer

knowledge about the system. However, there has been very little research into how the users’ or

the developers’ experiences and training affect this relationship. In our research we are hoping to

focus on the developer’s contribution to the user-developer relationship in ISD. By contributions

we mean when, how and why they seek user interaction during the ISD process. In this respect,

we are hoping to bring to light a little-studied aspect of developers’ contribution – how their

formal training, or the lack thereof, affects their perception and commitment to involving the

user in the development of an information system.

1.1 Relevance to IST

The research explores how technical knowledge and training affects developers’ relationship

with users. We hope to capture that information through exploring academic background,

experience, information system definitions, and user role beliefs. Our approach focuses on all

three constructs of the IST research philosophy – information, technology, and people. The ISD

process, though concerned with the developed of technological systems, is a social process

shaped by human actors. Developing an information system not only requires developers and

users to have knowledge about the domains of design and use of the system, but it also requires

effective information exchange between them. Our research attempts to shed light on the

2

dynamics of the developer’s role in user/developer interactions and points of influence for

improving those interactions. These findings will also highlight areas for future educational

opportunities and research collaborations between social and technical research within the realm

of Information Sciences and Technology.

The individual research interests of research team members cover a broad spectrum of issues

related to information systems development. Other than interest in user involvement in ISD, we

are interested in studying implementation of large-scale information systems, ontologies used to

capture users knowledge in ISD, and re-skilling of IT professionals into the information sector.

1.2 Roadmap

The following research proposal will clarify the contributions of the developer-user interactions

within the area of Information System Development. Section two will highlight the current

literature in the area of developer-user interaction. Section three will clarify the research

questions pursued, identify the methodological approach taken, reasoning behind the methods

employed, data collection implemented, issues that arose during the research process and roles of

the researchers. Section four examines the data gathered though coding, analysis of that data and

the findings. Section five summarizes the findings, implications and contributions the research

adds to the developer-user research area. Section six summarizes the research in its entirety.

2 LITERATURE REVIEW

User participation and involvement in the information systems development process has been a

long running theme in information systems research. User participation refers to the activities

performed by the user during systems development while user involvement is the importance and

personal relevance of the system to the user (Barki & Hartwick, 1989), (Barki & Hartwick,

1994), (Hartwick & Barki, 1994). Grudin (1991) argued the importance of users’ participation in

3

design, and suggested a classification for the term user (i.e. end-user, customer, client, etc.).

Even though users’ involvement and participation in the ISD process has been studied for over

four decades, it continues to present challenges for future research. Much attention has been paid

to issues related to users’ participation and involvement in the process of developing systems.

Several researchers (Bailey & Pearson, 1983), (Doll & Torkzadeh, 1989), (Hwang & Thorn,

1999), (Robey & Farrow, 1982), (Tait & Vessey, 1988) have concluded that user participation

significantly increases system success and end-user satisfaction. Other researchers, (Ives, 1984),

(Barki & Hartwick, 1994), (Butler & Fitzgerald, 1997), (Cavaye, 1995), have found that user

participation may not contribute much to improving the quality of the system being developed.

Most literature in this area refers to the relationship between users and developers as user

participation or user involvement. For the purposes of this research, we use the term interaction,

referring to a broader idea that covers both participation and involvement issues (Grudin, 1991),

(Rosove, 1969).

Empirical studies have also failed to reach consensus on whether user involvement ensures

success of the ISD process. Ives et. al. found no results supporting the correlation while

Pettingell et al. found a fair to moderate correlation (Emam, 1996). Wong’s case study analysis

of ETHICS versus a successful IS developed without structured theory found correlation

between users and success of ISD’s (Wong & Tate, 1994). Iivari’s Case Study analysis (2004)

suggested that users’ involvement could be influenced by the organization’s culture.

User participation should focus on users’ knowledge to achieve more effective design (He,

2004), (Wilson ET. Al., 1997). Users accumulate rich application domain knowledge due to long

period of exposure to their job context and “the knowledge perspective of user participation

captures the quantity (how much) or the frequencies (how often) of various knowledge activities

4

user participants perform during an ISD process” (He, 2004, p. 2). While He’s research addresses

the knowledge interaction between users and developers, our research will focus on the

developers’ perception of when the users’ knowledge is needed during the ISD phases and how

their formal training, or the lack thereof affects this perception.

Kautz & Pries-Heje (1999) and Nosek (2001) highlight that little research has focused on

education and training of systems development methodologies. Nosek suggests “education

involves understanding abstract theory while training involves the accretion of skills necessary to

perform a task” (p. 205). Roberts (2000) points out the importance of education for developers.

However, he states that some capable developers have achieved huge accomplishment with no

formal education. Roberts (2000) also suggests that “formal education in computing is neither a

necessary nor a sufficient predictor of productivity in the software development industry” (p.

85).

This issue with developers’ lack of training has been stressed by Rosove (1969), who

acknowledges the need for improving developers training, and updating computer sciences

programs. He argues that recent graduate students are weedy in software development techniques

because computer science programs emphasize operating systems, programming languages, and

some applications skills. Rosove also recommends that developers should be educated to work

and to interact with users, as well as to design systems for them. However, training has little

influence in adopting a system development methodology (Kautz & Pries-Heje, 1999). A recent

study (Latzina, 2002) has shown that developers’ training in usability still poses an important

issue on how developers involve users in ISD.

2.1 Preconceptions

The preconceptions applied to this research are based on the authors’ professional and academic

5

experiences. In terms of academic experience, two researchers have a Computer Science

background, and one researcher has a Management Information Systems background. Thus, we

have all been educated and trained in software engineering, and have been exposed to the ideas

and concepts related to ISD as part of our formal education. These preconceptions guided our

research approach with respect to defining the research questions, sampling selection,

elaborating the interview questions, performing the interviews, and analyzing the data. Based on

our own academic backgrounds and the common ground we shared, we initially started with the

assumption that developers who had courses in software engineering or ISD methodologies

qualified as “formally trained”. Using this criterion, we decided to consider undergraduate

degrees in computer science (CS) and information systems as “formal” training for ISD.

Developers with undergraduate degrees in all other fields were considered “informally” trained

or self-taught. The term “formal training” as used by us relates to formal education as defined by

Roberts (2000) to mean any completed undergraduate degree.

The interview questions, similarly, reflect our understanding of the IS development phases, and

the importance of users participation and involvement in systems development. We prescribe to

He’s (2004) view that the user has valuable contributions to make in the ISD process and hence

it is important for developers to involve the user in ISD.

6

3 METHODOLOGY

3.1 Research Questions

Developer’s styles of user interaction differ based on how they acquire technological skills

related to ISD. Formally trained developers’ tend to have very structured views of user

interaction based on development methodologies defined in IS literature between 1967 to 1977

(Fitzgerald, 2000). Informally trained developers have possibly been formally trained in an

entirely different subject area and then migrated to an ISD role. Thus, we were interested in

answering the following research question:

How does developers’ style of user interaction differ based on whether the developer is

formally or informally trained?

Two related sub-questions that help in answering the above research question are:

During which phases of the ISD processes do developers seek users’ knowledge?

During which phases of the ISD processes do developers feel that the users’ knowledge is

highly needed?

In order to answer our main research question, we chose to interview two different groups of

information systems developers, formally trained and informally trained developers as per our

assumptions above. As we conducted the research, certain issues and concepts emerged which

forced us to change our initial definitions of “formal” and “informal”. We discuss these issues

and the evolving nature of our understanding of the terms formal and informal later in Section

3.4.4.

We expected the outcome of this open-ended interview process would help us answer several

specific questions related to developers’ beliefs of user interaction in the ISD process. From the

7

first group of interviewees, we expected to learn if there were differences between what is taught

in school regarding the importance of user interaction and what is practiced by developers on a

daily basis regarding capturing and applying users’ knowledge. From the second group of

interviewees, we expected to learn how and why they became developers, where they received

their ISD training, and how they capture and apply users’ knowledge in an ISD process. In

addition, we also expected to learn what both groups considered as ISD phases, and how

important they considered user interaction in ISD.

3.2 Rationale and Epistemology

We analyzed both qualitative and quantitative research approaches to decide which methods

were better suited to answer our research questions and the overall nature of the phenomenon we

were interested in studying.

Using quantitative research methods would have required us to quantify the phenomenon we

were interested in studying by representing numerically the levels of our theoretical constructs

and concepts (Straub, Gefen, & Boudreau, 2004). Qualitative research methods, on the other

hand would enable us to examine phenomenon by understanding people and the social and

cultural contexts within which they exist though interviews, participant observation or document

analysis (Myers, 1997). Since there is no theory of how user involvement is viewed by formally

and informally trained developers, we could not perform theory-testing or quantify the constructs

that we thought were significant in this area. Thus, we decided to approach this research by

examining the social actors’ (systems developers, in this case) perceptions and ideas about the

significance of user interaction.

The most important factors that influence the choice of qualitative methods include the nature of

the research problem, the researcher’s theoretical lens, the degree of uncertainty surrounding the

8

phenomenon, the researcher’s skills, and academic politics (Trauth, 2001). To ensure that our

research was well suited for qualitative analysis we explored those factors.

In examining user interaction in information systems research, we did not have any a priori

hypothesis about how formally and non-formally trained developers differ in how they perceive

and practice user interaction. We were interested in examining and understanding a socially

based phenomenon. Thus, the ontological perspective we adopted was concerned with social

relations, processes and practices and was well matched to a qualitative research methodology

(Mason, 2002).

We could have chosen a positivist, interpretivist or a critical theoretical lens. The positivist

ontological assumption is that there is an objective physical and social world that exists

independent of the actions of humans. Positivist studies assume the existence of “a priori fixed

relations within phenomenon which are typically investigated with structured instrumentation”

(Trauth, 2001; p. 6). In the domain of information systems research, such studies have attempted

to test theory, hypothesis, and formal propositions in order to draw inferences about phenomenon

from the sample to the stated population (Orlikowski & Baroudi, 1991). Interpretivism is based

on the ontological assumption that reality and our knowledge of it are socially constructed.

Interpretive studies assume that “people create and associate their own subjective and

intersubjective meanings as they interact with the world around them” (Trauth, 2001; p. 6). In IS

research, interpretive methods are aimed at understanding the context within which an

information system exists and how it influences and is influenced by this context (Walsham,

1993). Critical studies aim to critique the status quo of deep-seated structural contradictions

within social systems with an aim to transform alienating and restrictive social conditions

(Trauth, 2001). Critical studies in ISD consider information systems in their wider social context,

9

focusing on issues such as power, domination, conflict and contradiction (Howcroft & Trauth,

2004).

The users’ minor role in IS development has been due to an underlying assumption that there is

an objective truth in organizations that can be mapped to develop successful information systems

(R. M. Wong & Tiainen). This positivist perspective has ignored the view that information

systems development is a social process shaped by people, both users and developers. We

embraced this interpretivist perspective in order to understand the importance of user interaction

in the process of ISD. The investigation of social phenomena should be interpretivist in its

orientation so that contextualized meaning can be revealed to enable socially-based phenomenon

to be understood (Butler & Fitzgerald, 1997 ). The interactions between users and developers are

socially-based phenomenon. We were interested in understanding what “user involvement and

participation” means for systems developers and how their understanding of the term differs

depending on their level of formal training, therefore we used an interpretive lens.

In this case, there was a high degree of uncertainty surrounding the research problem as very

little research explored the relationship between developers’ formal education and their

perception of the importance of user interaction. Past research in this area has stressed how user

interaction results in increased levels of user satisfaction and perceived usefulness of the system

(Terry & Standing, 2004). However, research regarding the relationship between user interaction

and success of information systems development is not grounded in theory nor substantiated by

research data (Butler & Fitzgerald, 1997 ).

The last factor pertinent to this research is the researchers themselves. Researchers familiar with

qualitative research methods are more likely to use them. Members of the research team had past

experience with qualitative interviewing, focus groups, and content analysis and felt comfortable

10

to use qualitative approaches for this study.

3.3 Research Design

A methodological strategy is the logic by which one goes about answering research questions

(Mason, 2002). Having decided on qualitative methods for investigating the phenomenon of

interest, we attempted to link our research questions and rationale to research methods and

practicalities. We adapted the table from (Mason, 2002) and linked her criteria with the research

rationale described above to help us decide on a research method.

How does developers’ Interviews of developers Attempt to discover Need two groups
style of user difference of opinions of ISD
interaction differ based Other Possibilities: between developers developers:
on whether the observation, interviews based on their formally and
developer is formally with users acquisition of informally
or informally trained? technology skills. trained.
During which phases Interviews of developers Attempt to elicit Need access to
of the ISD processes personal standards of diverse pool of
do developers seek Other Possibilities: user interactions from ISD developers
users’ knowledge? observation, document developers.
analysis
During which phases Interviews of developers Attempt to elicit Need ISD
of the ISD processes when developers developers who
do developers feel that Other Possibilities: believe that users’ have worked on
the users’ knowledge observation, document knowledge is needed. ISD portions with
is highly needed? analysis users involved.
Table 1: Developing a research design (Mason, 2002)

Our ontological position suggested that developer’s knowledge, formal training, views, and

experiences shape their perception of how important user involvement is to the process of ISD.

Our interpretivist epistemology lead us to believe that via interviews we could construct the

social reality of the phenomenon under study by getting first-hand accounts of how developers

feel their training in formal or informal fields affects how they treat users in the process of

developing systems. Talking to developers and observing their attitude towards user involvement

11

enabled us, as researchers, to be active and reflexive in the research process (Mason, 2002).

The inclusion criterion for subjects in this study was that they should have been involved in the

development of at least one information system, either during the course of their education (say

as a graduate student research project) or in the course of working in industry. An information

system, in general, consists of software, hardware, databases, servers, online services and

integration tools. While this definition of information system definitely qualifies for the purposes

of our research, we would also consider a software system or tool as an information system. An

information system development life cycle for this study is composed of planning, analysis,

design, implementation, and testing phases (Dennis et al., 2005), (Maciaszek, 2001).

We were interested in interviewing a total of twelve subjects, six of whom were formally trained

in ISD practices and techniques and another six informally trained. “Formal” training meant

undergraduate degree in Computer Science (CS). Subjects with undergraduate degrees in all

other fields were considered “informally” trained. These developers were self-taught in that they

never had courses in software engineering or development techniques but had picked up ISD

skills through books or online resources. Due to ease of accessibility, the subject included four

current graduate students at the College of Information Science and Technology, three Penn

State University Libraries staff and one Penn State Information Technology Services staff.

Using a semi-structured interview method, we asked subjects open-ended questions related to the

above mentioned themes. Talking to developers about how their training, formal or informal, has

shaped their practice of user involvement in the ISD helped us to explore these issues in their

social context.

12

3.4 Data Collection

This research followed Mason’s suggestions for planning and conducting qualitative

interviewing (Mason, 2002). She suggests we employ qualitative interviewing when we “want to

give maximum opportunity for the construction of contextual knowledge” (Mason, 2002, p. 64)

based on the participants’ experience. In this research, we were constructing knowledge about

participants’ perception of user interaction by examining how developers’ training has

influenced their ISD experience. An example of “constructing knowledge” is that instead of

asking the interviewees to sort a given list of ISD phases, we would rather expect them to

provide a list of ISD phases.

3.4.1 Subjects
We designed the study to include twelve participants, six of whom were formally trained in ISD

and the other six are informally trained. They were contacted via email (see email script in

Appendix A), and asked to participate in semi-structured interview lasting no more than forty-

five minutes. All participants contacted to participate in our study were Pennsylvania State

University staff or graduate students, and colleagues of the authors. In addition, we sought

people with technical skills and at least a year of professional experience in ISD.

From the initial twelve emails sent out inviting people to participate in our study, ten replied

confirming their interest in the study, one decided not to participate, and one did not respond the

email. From these ten confirmed participants, we were able to schedule only eight interviews

because of some time constraints. Fortunately, based on the eight interviewees’ academic

background and professional experience, we were able to form even groups of interest. Taking

into consideration the academic background, we had four participants formally trained and four

participants informally trained. Considering professional experience, all participants had

13

experience with industry while four participants moved from industry into academia.

Apart from the specific requirements above, this research considered that any other variable,

such as age, gender, race, and nationality, would not be important for the scope of the research.

Nevertheless, we had a well-balanced number of participants with different characteristics (men/

women, young/old, and different nationalities).

3.4.2 Interview Questions
The interviews we designed for this research (see Appendix B) consisted of open-ended

questions related to three main themes – participants’ education and skills, participants’

perception of what the ISD process is, and participants’ perception of user interaction in ISD.

The first theme, regarding educational and skills, helped us to learn and identify the

interviewee’s background in systems development. Although we had a broad idea about what a

‘formally’ trained developer would be, we relied on our findings to identify what kind of formal

education actually influences the way developers perceive the user’s interaction in ISD. The

development process theme covered information about the interviewee’s knowledge about ISD

phases and sub-phases, as well as their relations with user’s participation and involvement in

ISD. We expected that people with different educational backgrounds would have different

opinions about the ISD phases. Lastly, the user participation and involvement theme would

indicate how important the interviewee feels user interaction is within the ISD phases. For this

theme, we were expecting to learn about the interviewee’s experience involving the user in

different ISD phases. Although, it is widely known that users’ participation in ISD should be

carried throughout all the ISD phases, we suspected that it would not happen in practice.

Each interview question aims to understand and to cover a specific category within the themes.

These categories are the initial source for our coding schema (see Appendix C).

14

3.4.3 Interview Process
Each interview lasted approximately forty-five minutes, and was performed by at least two

interviewers. One interviewer was responsible for conducting the interview and asking the open-

ended questions. The other interviewer was responsible for taking notes, although, the interviews

were also audio recorded. The second interviewer could also contribute with questions,

comments, or clarifications.

After each interview, the interviewers got together to discuss possible issues or suggest

corrections for the next interview. Sub-section 3.4.4 presents a summary of the main issues

found during the interview process.

The environment, where the interviews took place, performed an important role for the

accomplishment of the interviews. The collaborative space room was considered by the group as

the best place to conduct interviews. It provided us with a quite place, and an excellent layout

for talking to the interviewees. The interviewees’ office was relatively quite, except for some

interruption of their officemates. Lastly, the interviewers’ laboratory was the worst place to

interview people because it provided a significant level of distraction to the interviewee. The two

laboratories used to do the interviews are shared with other graduate students, and consequently

caused too much noise.

3.4.4 Issues
The first interviews helped us to identify issues such as missing questions and topics that are

relevant to the information we are seeking in this research. Instead of asking participants about

their proficiency in computer languages and development methods/tools, it was more useful to

ask them about their experience with the same. Based on the first interviews answers, we noticed

that the term “proficiency” implied the participant’s current ability with a technology. On the

15

other hand, the term “experience” implied all abilities (present and past) with a technology. By

switching the terms, we were able to generate more data about the participant’s experience. The

sequence in which the interviewer asked questions was also changed to allow the conversation to

flow smoothly, as well as to help the interviewees to better remember their experiences. For

instance, we noticed that first asking the interviewees to list their experience with programming

languages and development methodologies would help them to trigger their memory to

remember how they learnt their skills. Asking the other way around was making the interviewees

to spend extra time retrieving their memory in order to provide an accurate answer.

During the interview process, we also came across some issues resulting from the authors’ lack

of experience with conducting interviews. First, because some interviewees provided vague

answers to the questions asked, we noticed a tendency for interviewers to lead the participants’

answers. A possible explanation for that is because the questions were not well formulated and

led to multiple interpretations, or the interviewee was not elaborating enough and the answers

were not producing the adequate information. Second, some interviewees had the tendency to

speak a lot, and consequently dominated the interview. In these cases, we witnessed the tendency

to add extra information to illustrate points, usually extending the conversation, and introducing

many facts (relevant and irrelevant). Guided by our curiosity and the expectation of some useful

data, we usually let the interviewee talk until we felt that we were not getting anymore relevant

information. It was a challenge for the interviewers to interrupt the conversation and get it back

on track. Third, some interviewees suggested us to hand out the questions in advance, so they

would be better prepared to answer them. Having the questions in advance would help the

interviewees to go through their past professional and academic experiences, and to remember

relevant facts to answer the interview questions. Prior knowledge of the questions could also

16

sway the position of the interviewee in the realm of user interaction and its importance. Finally,

some interviewees got easily distracted, and started to give us irrelevant information. For

instance, an interviewee became interested for one of the author’s MAC laptop, and started to

ask about its features.

Another significant issue was defining where to draw the line between ‘formally’ and

‘informally’ trained developers. Our initial idea was to define people with degrees in computer

science and information systems as formally trained and all others as informally trained. This

presented a philosophical difficulty because although CS and IS programs may have some

overlap in terms of the courses taught, the fields themselves have different underlying

philosophies with respect to the importance of the user in the development process. This could

mean that ISD training imparted in these fields is very different and hence we need to redefine

our notion of ‘formally’ trained developer. Therefore the number and the type of such courses

typically taken by participants’ from CS and IS helped us to define ‘formal’ training for the

purposes of this research.

3.5 Role of Researchers

All members of the research group took an active interest in all aspects of this study and worked

well as a team. Through regular meetings members of the team discussed the research and the

findings as they emerged, as well as contributed equally to writing reports, recruiting

participants, and conducting interviews. Each researcher recruited four participants, conducted at

least two interviews, took notes of at least two interviews, and coded four interviews. The

approaches provided us with valuable opportunities to learn and perform different tasks in a

qualitative research process. More information about the role of researchers can be found in sub-

sections Preconceptions, Interview Process, and Data Analysis.

17

4 RESULTS

4.1 Data Analysis

Data analysis began during the first interview and continued to refine itself throughout the entire

data collection process. Since qualitative methods were employed, much of the analysis

included the researchers internally processing the information and then elaborating their thoughts

during meetings and across email. The analysis process included physical processing of data,

data categorizing and coding, and analysis.

4.1.1 Processing of Data
Physical processing of data included two main tasks: reviewing and transcribing interview audio

and integrating additional researcher interview notes. During interviews one researcher was

appointed interviewer and responsible for ensuring adequate coverage of all the main themes of

our interview guide (see Appendix B). That researcher was also responsible for the interview

transcription. This ensured that the interviewer gained a full working knowledge of the interview

since they were typically slightly distracted by formulating questions and exploring additional

opportunities the interviewee may have mentioned.

The other researchers present for the interviews were responsible for note taking and then

integrating those notes into the appropriate location within the transcripts. This was useful for the

coding and analysis because it highlighted when and where areas of researcher enlightenment

became apparent within the process. The revisiting of interviews and notes was an important

factor in developing the researchers’ intimacy with the data.

4.1.2 Categorizing and Coding
We were interested in understanding what “user involvement and participation” means for

systems developers and how their understanding of the term differs depending on their level of

18

formal training, therefore we were using an interpretive lens to understand the data. Mason

cautions that the main issue with using an interpretive lens is representing research participants’

views rather than imparting your own views onto the data they provide (Mason 2002, p.76). In

order to counteract that tendency, we individually formulated our thoughts on emerging concepts

and then held group meetings to check our approaches.

During the research group meetings specific categories of interest became evident. Some of the

categories were driven by literal data such as work experience, years, type of training while

others were more interpretive in nature; user importance. Eleven high-level categories emerged

from the group meetings and were utilized to create data codes. Those categories are described in

detail in Appendix C. The allocations of interview participants in the identified categories are

presented in Table 2.

19

Category Codes Participant Results
Academic The participants included four formally trained with Computer Science
Background degrees (two recent graduates, two experienced graduates) and four
informally trained with degrees in Business, Engineering, Geology and Hotel
Restaurant & Management.
Acquired Skills All participants cited self-teaching as main approach to skill acquisition.
Half of the formally and informally participants gained skills through
coursework. One formally trained participant utilized all three venues for
skill acquisition.
Languages Only two of the formally trained participants had both types of experience
while two of the formally and one informally trained participant had low-
level programming experience and three informally trained participants had
script/web experience.
Information Unfortunately it was not until the researchers were half way through the
System interview participants that it emerged as potentially important. For the
Definition participants captured (two formally and three informally) only one of the
formally could defined an IS and all of the informally had working
definitions.
Information The participants typically cited a waterfall or cyclical approach with only one
System Phases formally trained participant unable to identify any phases to the ISD process.
Career Path Four of the five graduate experience developers also had experience in the
software industry while two of the four information technology experienced
developers started their career in industry other than technology.
Knowledge Three of the four direct type developers were also informally trained while
Elicitation three developers were indirect and one developer had no interest in
knowledge elicitation from users.
Past Participation Two of the four formally trained and two of the four informally trained
in ISD developers participated in all phases they identified.
Past User In one extreme case, a developer admitted, without a bit of remorse, that
Interaction when lacking appropriate details about requirements they would simply role
play scenarios of the system in their head and then use their best guess to fill
in the requirement gaps.
Participant’s A majority of the participants did not see a difference between ideal and
Ideal User current ISD, the few that would change the interaction were typically
Interaction involved in low-level programming and would only change if the project
necessitated user interaction.
User Importance The participants were fairly dispersed along the levels of user importance:
one None, three Low, one Medium, and three High level.
Table 2: Applying Code Schema to interviews

After we identified the eleven categories, two researchers applied the categories to the results at a

20

high level (Figure 1).

Figure 1: High-Level Interview Coding

Independently, the third researcher mapped the categories and codes to the transcripts and

integrated notes (Figure 2). The qualitative software package Atlas was employed to perform

content analysis of the interview transcripts. The low-level codes assigned during the content

analysis were later mapped to the high-level codes developed by the other two researchers. This

approach ensured corroboration of results by two independent coding techniques and helped

decrease individual researcher biases that may have influenced data interpretation.

21

Figure 2: Atlas Coding

22

4.1.3 Analysis
Through categorizing and coding of the interview data, three main factors emerged as promising

determinants of developer’s user interaction beliefs: Academic background (formal and informal

training), experiences and access to users. To analyze these factors we looked at the interpreted

user importance levels versus the individual factors with the assumption that user importance

levels are key to determining developer-user interaction during the ISD process.

Examining formal and informal training versus our interpreted user importance illustrates that

formal training (Computer Science) spread evenly across the importance levels while informally

trained users had a higher level of user importance.

High

Medium

Other
Low
Computer Science

None

0 1 2

# of Participants

Figure 3: Academic Background vs. User Importance

This comparison leads us to believe that academic experience plays a smaller role than we

originally assumed. This does not mean that academic experience plays no role in the developer-

user interaction. There is also notable increase in user importance with participants that received

an informal training in Information System Development. This may be due to their own

experiences prior to becoming a developer.

The second factor, developer’s experience, refers to the languages the developer has experienced

23

and the career path they took to get there. Two of the formally trained developers’ experience

consisted of low-level programming and graduate work while the other two had experience in

both low-level and script/web programming. All four of the informally trained developers had

strictly script/web programming experience.

High

Medium
Both
Low Script/Web
Low-Level

None

0 1 2
# of Participants

Figure 4: Experience vs. User Importance

Low-level developer’s tended to have a lower level of user importance while the others had a

high level of user importance. We believe this tendency of Low-level developer’s was a product

of their work environment. The Low-level developers focused on more system based

programming while the Script/Web developers focused on high level applications with a higher

level of usability for gathering and displaying information. While users utilize system based

programs they do not typically interact and extract information from those programs. System

based programs are more task-driven than information distribution driven.

24

The final factor, developer’s access to user, refers to the level of insulation between the

developer and the user during the ISD process. Throughout our interview and data collection

process it became apparent that user interaction beliefs hinged on the role of the developer within

the ISD process. If the developer obtained the user information indirectly (through a third party)

then the level of importance was typically lower. We believe this was due to the insulation

between the developer and user. Since the user is less prominent in the developer’s work, they

are not as visible and therefore not deemed as important.

High

Medium

Direct
Low
Indirect

None

0 1 2

# of Participants

Figure 5: User Knowledge Access vs. User Importance

4.2 Findings

Our main research question was how developers’ style of user interaction differs based on

whether the developer is formally or informally trained. We found that developers’ academic

background played a small role in how important they perceived the user to be. In shaping

developers’ perception of the importance of user involvement on-the-job experience and work

context were seen to play a more important role than their academic background and education.

Specifically, we found that developers with undergraduate degrees in computer science did not

25

perceive user involvement to be as important as did self-taught developers. This shows that

formal education in software development does not emphasize the importance of user

involvement in ISD. Self-taught developers perceived user involvement to be highly important

mainly because their organizational context and work experience had taught them the importance

of user involvement.

Developers’ perception of users and the importance of interacting with them during the

development of an information system were highly influenced by their work context. The

organization’s work culture, development philosophy, freedom and constraints placed on

developers with respect to resources and access to users were found to influence the way

developers actually involved the user to a much larger extent than what they may or may not

have learned through their formal education.

We also found that developers working with Web-based or scripting applications perceived the

user to be more important than programmers who write low-level code. Butler (2003) makes a

distinction between Web-based information systems and “traditional” information systems with

respect to the social and organizational issues associated with their development and

implementation. Along those lines, we found that developers who write Web-based or scripting

applications are more likely to perceive user involvement as important. It could be due to the fact

that Web-based applications are more interactive and user satisfaction is much higher when the

user is highly involved iteratively in the development process.

Finally, the access to the user or “level of insulation” between the developer and the user affects

developer-user interaction. The higher the “level of insulation” the developer has from the user

the less important user interaction is for the developer. By “level of insulation” we mean

organizational hierarchy which prevents the system developer from coming in direct contact with

26

the user for knowledge elicitation purposes. We found that low-level programmers were

“insulated” from the user and obtained user requirements and needs via other members of the

development team who actually interacted with the user. The organizational hierarchy prevented

them from directly talking to users. Therefore, when they perceived insufficient or incomplete

information from the user, they were unable to go back to the users and get more information. In

this case, one formally trained developer said he would “make assumptions about how the

common user is going to use” the system. Even then, being an expert, it was hard for him to

imagine how a user without technical-knowledge would use the system he was designing. This

example highlights that work context and organizational constraints influence how far

developers involve the user much more than formal training does.

5 DISCUSSION

In summary, developer’s training, whether it be formal or informal, plays only a small role in

determining developer’s beliefs in user interaction within the ISD process. Work experience and

context play a much larger role than academic experience because developers must conform to

their current working environment. Developer’s in script and web based ISDs tend to interact

with the user more which we believe increases their likelihood to deem the user as important to

the system development. Lastly, the level of access to the user also plays a factor in user

importance and therefore user interaction seeking because of the nature of low-level

programming.

5.1 Implications

The findings of our study have important implications for education and industry. They make us

question the role of education in making systems developers aware of the importance of user

involvement. Our research illustrates that developers with undergraduate degrees in computer

27

science did not receive any formal courses that stressed the importance of involving users. These

formally trained developers were typically recruited in coding jobs. Among them, the low level

programmers typically did not learn the importance of the user because of their insulation from

them. This raises concerns about the inadequacy of traditional computer science curricula to

address the need for teaching developers about the importance of user involvement. Computer

science curriculum studies are still slow in embracing user-centered development methods

(Kaasinen & Clarke 1998). User-centered design techniques should be taught as part of

traditional CS courses as they stress on involving users throughout the development phases.

According to Rosson & Carrol (2001), usability has been studied for the last three decades, and

become a focal point in software development, especially in requirements analysis and interfaces

design. Seffah & Andreevskaia (2003) found that most of systems analysts have little or no

training in usability techniques and suggested that developers should be made aware of user-

centered techniques.

Also, our study raises concerns that organizations hiring computer science graduates are hiring

systems developers who do not necessarily perceive users to be important to the process of ISD.

Consequently, much of the usability training and education for developers is left to corporate

training organizations (Kaasinen & Clarke 1998). This may lead to products being developed

which have low usability and lead to low level of user satisfaction unless the organization has a

culture of promoting user interaction in ISD. Thus, if organizations are interested in hiring

developers who perceive users as important, they will have to either hire developers who have in

their past job-experience treated users as important or train them to consider the same.

5.2 Contributions

Past research into importance of user involvement in ISD has looked mostly at end-user training

28

as important for the success of information systems. However, very little research has looked at

how developers’ formal educational background and training influence their perception of user

interaction. This research highlights potential factors for future research within the realm of

developer examination. It also illustrates that current educational efforts are not determining the

ways developers choose to interact with users.

5.3 Limitations

Due to time and resource constraints, our study has several areas for improvement. Even though

we were interested in studying the differences between formally and informally trained

developers, we struggled with the issue of defining “formally trained” developers throughout our

research. We started out considering that developers who had courses in software engineering or

ISD methodologies qualified as “formally trained”. Using this criterion, we decided to consider

undergraduate degrees in computer science and information systems as “formal” training.

However, the underlying educational philosophies of computer science and information systems

fields with respect to the perception of the user differ considerably. Also, as we spoke to

computer science graduates, we realized that most of them had not had any courses in software

development methodologies or software engineering. Thus we changed our initial definition of

formally trained to include only computer science graduates as they had formally learnt ISD

skills as opposed to all other degree-holders who had learnt ISD skills on their own.

Also, our study used a small sample of eight developers and the sample could be considered to

be biased as most of the developers we talked to belonged to the same organization. The

graduate students were mostly from the College of Information Sciences and Technology and the

developers in industry were Penn State University Libraries or Information Technology Services

staff. Since we found that work context plays a large role in perception of the user, using a more

29

diverse sample of developers would have lent more credibility to our research.

5.4 Future Research

Our research highlights a gap in IS literature with respect to the developer’s role in user-

developer interactions during the ISD process. In the future this research can be extended by

using a larger sample size and talking to developers from diverse organizations. The distinction

between Web-based and traditional developers with respect to their difference in perception of

the user needs to be studied in more detail. Some unexpected issues emerged from our research

other than the ones we were interested in addressing. We found that the age of developers

affected their perception of user interaction. Older formally trained developers said that their

education was so long ago that it had ceased to have much of an influence on their career today.

Also, computer science degrees in the 70s and 80s did not include software methodology

courses. Thus, older developers tend to be more influenced by their on-the-job experience than

younger developers whose education is still fresh in their memory to be applied in their work.

6 CONCLUSION

Our research preliminarily highlights a gap in the area of developer-user interactions. Although

our sample group was small, we believe it was a significant source of evidence towards the

determining factors in developer’s user interaction beliefs. The three main factors that emerged

in this research were developers’ academic background, experience and level of insulation. With

further exploration of these factors, we believe that valuable insights can be revealed to aid with

future educational, employment, and research efforts.

7 REFERENCES

Bailey, J. E., & Pearson, S. W. (1983). Development of a tool for measuring and analyzing
computer user satisfaction. Management Science, 29(5), 530-545.

30

Barki, H., & Hartwick, J. (1989). Rethinking the Concept of user Involvement. MIS Quarterly,
13(1), 53-63
Barki, H., & Hartwick, J. (1994). Measuring User Participation, User Involvement, and User
Attitude. MIS Quarterly, 18(1), 59-82.
Butler, T., & Fitzgerald, B. (1997 ). A case study of user participation in the information systems
development process. Proceedings of the eighteenth international conference on
Information systems, Atlanta, Georgia, United States
Cavaye, A. L. M. (1995). User Participation in Systems Development Revisited. Information and
Management, 28, 311-323.
Doll, W. J., & Torkzadeh, G. (1989). A Discrepancy Model of End-User Computing Involvment.
Management Sciences, 35(10), 1151-1171.
Emam, K., Quintin, S., Madhavji N. (1996). User participation in the requirements engineering
process: An empirical study. Requirements Engineering, 1(1), 4.
Fitzgerald, B. (2000). Systems Development Methodologies: the Problem of Tenses. Information
Technology & People, 13(3), 174 -185.
Grudin, J. (1991). Interactive Systems: Bridging the gaps between Developers and Users. IEEE
Computer, 24(4), 59-69.
Hartwick, J., & Barki, H. (1994). Explaning the role of user participation in Information System
Use. Management Science, 40(4), 440-465.
He, J. (2004). Knowledge Impacts of User Participation: A Cognitive Perspective. Paper
presented at the SIGMIS, Tucson, AZ, USA.
Howcroft, D. and E. M. Trauth (2004). The Choice of Critical Information Systems Research.
Relevant Theory and Informed Practice - Looking Forward from a 20 Year Perspective
on IS Research. Boston, MA, USA, Kluwer Academic Publishers
Hwang, M. I., & Thorn, R. G. (1999). The Effect of User Engagement on System Success: A
Meta-analytical Integration of Research Findings. Information and Management, 35,
229-236.
Iivari, N. (2004). Enculturation of User Involvement in Software Development Organizations:
An interpretative Case Study in the Product Development Context. ACM NordCHI,
Finland, 287-296.
Ives, B., Olson, M. (1984). User Involvement and MIS Success: A Review of Research.
Management Science, 30(5), 586-603.
Kautz, K., & Pries-Heje, J. (1999). Systems Development Education and Methodology
Adoption. ACM SIGCPR Computer Personnel, 20(3), 6-26.
Latzina, M., & Rummel, B. (2002). Collaboration-Based Usability training for Developers.
Mensch & Computer 2002: Vom interaktiven Werkzeug zu kooperativen Arbeits- und
Lernwelten. Stuttgart: B. G. Teubner, S. 285-291.
Mason, J. (2002). Qualitative Researching (2nd Edition ed.). London, UK: Sage Publications.
Myers, M. D. (1997). "Qualitative Research in Information Systems." MIS Quarterly 21(2):

31

241-242.
Nosek, J. T. (2001). Justifying the Business Value of Information Systems Education: A Report
on Multicultural Field Experiments. SIGCPR, San Diego, 205-211.
Orlikowski, W. J. and J. J. Baroudi (1991). "Studying information technology in organizations:
Research Approaches and Assumptions." Information Systems Research 2(1): 1-28.
Roberts, E. (2000). Computing Education and the Information Technology Workforce. SIGCSE
Bulletin, 32(2), 83-90.
Robey, D., & Farrow, D. (1982). User Involvement in Information Systems Development: A
Conflict Model and Empirical Test. Management Science, 28(1), 73-85.
Rosove, P. E. (1969). Training Requirements for computer programmers in relation to system
development phases. SIGCPR, Illinois, 65-76.
Rosson, M. B., & Carroll, J. M. (2001). Usability Engineering: Scenario-Based Development of
Human-Computer Interaction. Morgan Kaufmann.
Seffah, A., & Andreevskaia, A. (2003). Empowering Software Engineers in Human-Centered
Design. IEEE International Conference in Software Engineering, 653-658.
Straub, D., D. Gefen, et al. (2004, January 7, 2005). "The ISWorld Quantitative, Positivist
Research Methods Website." Retrieved February 5, 2006, from
http://www.dstraub.cis.qsu.edu:88/quant/.
Tait, P., & Vessey, I. (1988). The Effect of User Involvement on System Success: A
Contingency Approach. MIS Quarterly, 91-108.
Terry, J. and C. Standing (2004). "The Value of User Participation in E-Commerce Systems
Development." Informing Science Journal 7
Trauth, E. M. (2001). Qualitative Research in IS: Issues and Trends. Hershey, PA, USA, Idea
Group Publishing.
Walsham, G. (1993). Interpreting Information Systems in Organizations. New York, NY, USA,
John Wiley & Sons, Inc.
Wilson, S., Bekker, M., Johnson, P., & Johnson, H. (1997). Helping and Hindering User
Involvement – A Tale of Everyday Design. ACM-CHI’97, Atlanta, 178-185.
Wong, E., & Tate, G. (1994). A Study of User Participation in Information Systems
Development. Journal of Information Technology 9(1), 51.

32

8 APPENDIX A

Email script for recruiting participants

Dear <Participant>,
We would like to request your voluntary participation in our study “Effects of
Developers’ Training on User-Developer Interactions in Information Systems Development”
(IRB# 22846). We believe that you are currently, or have in the past, worked on designing,
developing, implementing or maintaining an information system. We would like to interview
you about your experiences during this process, specifically about your involvement of the user.
The questions in the interview will be related to the following main themes: your
educational background, how you practice information systems development, where you learnt
the skills you use, and when and how you involve the user. The interview will be audio-recorded
with your permission.
The interview will take place at a time and location convenient to you and will not exceed
45 minutes.
This research may shed light on how you, as a developer, perceive user involvement as
an important part of the information systems development process. Also, this could lead you to a
better understanding of how your formal training in information systems development, or the
lack thereof, leads you to involve the user.
If you are interested in participating in this research, please reply to this email with
possible times when you would be available to participate in the interview. If you have further
questions about the research, please contact Andrey Soares at asoares@ist.psu.edu.
This research is associated with the Spring Semester IST541: Qualitative Research in IST
at The Pennsylvania State University.

Thank you for your consideration,
Jennifer McCauley, Sharoda Paul, and Andrey Soares

Sharoda A. Paul, IST PhD Graduate Student
323 Information Sciences and Technology Building
University Park, PA 16802
(814) 865-6846
spaul@ist.psu.edu

Jennifer McCauley, IST Master Graduate Student
102 Paterno Library
(814) 863-7098
jmccauley@ist.psu.edu

Andrey Soares, IST PhD Graduate Student
309 Information Sciences and Technology Building
(814) 865-6177
asoares@ist.psu.edu

33

9 APPENDIX B

Interview Questions Guideline
The following interview questions will cover three categories that we believe can help us
to identify and to reveal how developers: learn their skills, understand systems development
process, and perceive the user participation in ISD phases.

Education and Skills
• What is your academic background (i.e. Computer Science, Geography, etc)?
• How did you learn the skills to become a system developer (i.e. Associate Degree,
Bachelor Degree, Master Degree, PhD Degree, Technical/Professional Courses,
or Self)?
• Which programming languages and methodologies you have experience with (i.e.
Java, C, Perl, Visual Basic, O-O, E-R, etc)?
• What is Information Systems?
• How long have you worked developing systems for?
• List all job positions related to Information Systems Development that you have
already performed (i.e. Programmer, systems analyst, Project Manager, etc)?
Please indicate how long you spent working in each position.

Development process
• What are the phases and sub-phases of an Information Systems Development life
cycle (i.e. analysis, implementation, etc)?
• How do you learn about new suggestions and new implementations?
• Who is responsible for designing interfaces and reports? Do you ask for user’s
approval?
• How much time developers usually spent testing the system?

User Participation
• What is the importance of users’ involvement in IS development? Explain.
• Which activities of an ISD do users get involved in (i.e. requirements, etc)?
• How important user suggestions are to an ISD process (i.e. very important,
reasonable, and unimportant)? Which ISD phases?
• Are users usually involved in quality assurance?
• Have you worked with users for capturing their knowledge about the systems?
o If yes:
 How did you collect the user’s knowledge? Which ISD phases?
 How did you save their knowledge?
o If no:
 Who was the information source?
 How is the information captured?
 How was the information passed to you? In what format?
 Was the information given to you enough to understand the
system?
 Would you be able to explain how the system works?

34

10 APPENDIX C

Coding Schema

− Academic background - Undergraduate or graduate work within formal or informal
areas relevant to ISD.
o CS
o IS
o Business
o Engineering
o Other
− Acquired Skills - Acquisition of skills used to build ISD’s were acquired: coursework,
self-teaching, or training (sponsored by industry employer).
o Courses
o Self-training includes books, tutorials, web resources
o Required training
− Languages - Type of programming languages developer has utilized while participating
in ISD: Low-level programming (Drivers, kernel, operating system based) or
Scripting/Web based (Javascript, ColdFusion, etc) or both.
o Programming
o Scripting/web
o Both
− Definition of IS - This category was created to capture whether or not the participant
could define an information system as an entity.
o Can and what they are
o Can’t
− Career path - Capturing the path the developers took towards ISD experience.
o Information Technology (Primary & Secondary)
o Non-Technology
o Undergraduate and Graduate work
− IS phases – Meant to capture whether or not the developer could define specific phases
of development,
o Can and what they are
o Can’t
− Knowledge Elicitation - Within user interactions, vital information for ISD is elicited
through varying approaches, we categorized these into two types: direct and indirect.
The direct approaches involved the developer interacting with the user while the indirect
approach includes involvement of a third party to gather and then translate/communicate
requirements to the developer.
− Past Participation in ISD - This category sought to identify the level of experience the
developers had with ISD.
o Match to part of the definition of IS
− Past User Involvement - Designed to identify if participants actually practice what they
“preach”.
− Participant’s Ideal User Interaction - Politics motivate many areas of our lives and

35

development of ISD is no exception. This category was an opportunity for the
participants to identify their ideal user interaction level within ISD without regard to their
political situation.
− User suggestion importance - The most interpretive category in our research, user
importance was determined based on the researcher’s impression of the participant and
not the participant’s declaration of user importance.
o none (“I am the user”)
o low (“I know the user exists, but are not necessary”)
o medium (“When possible it is a bonus to involve the user”)
o high (“I start with the user and end with the user”).

36

Effects of Developers’ Training on User-Developer Interactions in Information Systems Development

More Related Content

What's hot (16)

Viewers also liked (6)

Similar to Effects of Developers’ Training on User-Developer Interactions in Information Systems Development (20)

Recently uploaded (20)

Effects of Developers’ Training on User-Developer Interactions in Information Systems Development