Case study of rules as relational data
1 like•348 views
The document presents a case study on an enterprise system developed by a company to represent business rules primarily as relational data instead of through traditional coded logic. The approach, termed 'ultra-structure,' allows for business rules to be managed directly by subject experts and greatly reduces the complexity and costs associated with software maintenance, thus enabling better adaptation to changing business needs. The results from a 23-year study show significant growth and efficiency improvements in the company's operations since implementing this system.
Case study of rules as relational data
- 1. Cover Page Case Study of an Enterprise System That Represents Rules Primarily As Relational Data Rather Than via Code Author: Jeffrey G. Long ([email protected]) Date: 2008 Forum: Acta Systemica Vol. 8 No. 2 (2008) pp. 47‐54 available at http://iias.info/pdf_general/Booklisting.pdf Contents Pages 1‐6: Preprint of paper. License This work is licensed under the Creative Commons Attribution‐NonCommercial 3.0 Unported License. To view a copy of this license, visit http://creativecommons.org/licenses/by‐nc/3.0/ or send a letter to Creative Commons, 444 Castro Street, Suite 900, Mountain View, California, 94041, USA. Uploaded June 24, 2011
- 2. Case Study of an Enterprise System That Represents Rules Primarily As Relational Data Rather Than via Code Jeffrey G. Long [email protected] Abstract Business rules are often narrowly defined, but they should include any and all rules that an organization follows to manage its affairs. These include all work processes, pricing rules, applicable laws and regulations (e.g. taxes), ontologies (e.g. defining the categories by which customers or products may be grouped), taxonomies (e.g. defining how a product or customer is actually categorized), and other kinds of rules. There is a distinct subset of these that are the technical interface rules which define various interfaces, such as with users (through forms and reports), data (via database management software), and other systems. In traditional software applications such rules are typically encoded as parameter-driven software. Because there are many rules, applications frequently require millions of lines of code. Such a large code corpus ensures significant life-cycle maintenance costs, as only programmers can update the code, and the code must change as the rules change over time. The approach we’ve used, called “Ultra-Structure,” is to remove any business rules that might ever change from the software, leaving only the control logic for a “competency rule engine” as software. All the rest of the rules are represented as data. We use and advocate a relational database approach to representing rules because of referential integrity, normalization, security, and other benefits offered by any modern RDBMS. Results are presented from a 23-year case study of a particular company. Key Words: Enterprise System; Databases; Business Rules; Software Economics; Business Process Reengineering Problems of Software Development One of the most vexing problems in the software industry is why over the past 50 years software (and software development) has not improved exponentially in its effectiveness and efficiency, in a manner comparable to hardware. While even home computer users can now afford a terabyte of data storage, writing code is nearly as laborious and error-prone as it ever was. In spite of decades of research and development, many software systems either do not or cannot meet users’ current or future requirements. Their construction and maintenance cost and time are also extreme. Solving this will not merely require that analysts ask better questions of users, for users can only say what they know now: how future government or other regulations, or mergers or divestitures, 1
- 3. or new ideas or technologies, may affect their requirements are inherently unknowable. Furthermore, user understanding of their requirements evolves, especially under the guidance of an analyst who asks about logically-possible alternatives to whatever the user posits. They can also evolve when an analyst identifies for them new ways that computers or other decision- making technologies might assist them. Solving this will also not merely require that programmers become more efficient, for what programmers encode is the rules of the business, which are subject to constant change. Current Integrated Development Environments and libraries of software classes can only address a snapshot taken at a given time; changing code in response to changing user requirements will always be required. And when business rules are dispersed over hundreds of thousands of lines of code, even a well-designed system poses maintenance challenges. If important user requirements are inherently unknowable and unpredictable, then how can any system ever be designed? Most other areas of engineering have been remarkably successful in predictably delivering results. But applications such as bridges and homes, while complex, are physical systems where user requirements tend to change little or within limited areas. The standard engineering approach that moves from analysis of requirements, to design, to construction of the system, is excellent for these situations. But knowing all likely user requirements is a luxury not available in software design. Nor should it be, for software captures thinking in ways that other technologies do not, and thinking will always evolve. Yet we must aim to design systems that can act “intelligently” and can change greatly over time. This can be done by representing rules in a different and better manner. It requires converting rules from their natural language form (e.g. as in a policy manual) into one or more rules in a canonical form; categorizing those rules into a small number of formats called “ruleforms” that are defined by their form and meaning, such that any logically possible rule pertaining to that application area (e.g. order processing) can be expressed somewhere in the system; and managing these rules as records in various tables (one per ruleform) through a relational database management system. Thus, business rules are represented not as software, and not as data in XML tags, but as records (relations) in a modern relational database management system, such that referential integrity and security of the rules is guaranteed, and information can readily be accessed through various tools such as queries and/or report writers. One key benefit of this approach is that subject experts (e.g. trained business managers) can read and manage rules directly, without having to first explain them to programmers. This can help greatly as companies try to better manage their knowledge resources by making them external to the subject expert. Another key benefit is that it facilitates changes to work processes. This can help greatly as companies try to better manage their work processes by adapting them to new technological, regulatory, competitive, or other constraints. In this approach software is seen as essential to the solution, but is not itself the solution. Instead software is a problem that must be eliminated whenever possible. 2
- 4. Case Study (1985-2008) The Company we are discussing here is a privately-held company which currently has about 50 warehouses throughout the United States. They are wholesalers of a commodity product, and are thus competing purely on price and service. Their customers are thousands of retailers who then resell the product to an end-customer who may be waiting in their office. The Company normally delivers orders within a few hours of order placement. It processes thousands of orders a day at its various branches, as well as from the Internet. In this competitive environment it has grown to become the largest independent wholesaler (i.e. not a manufacturer) in its industry. Intellinomics Era (1986-2002) Starting in September 1985, my former consulting firm (Intellinomics Corporation) built an enterprise system for the Company. At that time they had about USD $13 million in revenues. This system encoded as data only the interface rules regarding the format of screens and reports, user access authority, data structures, data retrieval, etc., while hard-coding in software the business rules such as work processes. The development cost of the system was about USD $250,000; it ran on a Digital Equipment Corporation VAX computer, had a character-based interface, and required about 35,000 lines of C code. It supported nine warehouses and about 50 users. The system was installed in September 1986 and was used by the Company for 16 years, until 2002. During this time the Company grew ten times as large, to about USD $130 million. It had no programming staff and used about ten hours per month of contracted programmer time. Industry-Leading System Era (2002-2005) The Company decided to move to a package that was the most widely used among wholesalers in the vertical market of the Company. Few if any requirements were discussed, and the system was installed in 2002. The vendor made many assumptions about how wholesalers in that vertical market would/should operate, but the Company had its own internal work processes and ontologies that were different. The system ran on an IBM minicomputer under AIX. It provided a character-based user interface and a highly unusual non-relational database system. After 18 months of working with the vendor, the system was deemed unacceptable. CoRE650 Era (2005-present) In March 2003 my current firm, CoRE650 Solutions LLC, started development of a new system. The Company requested the use of off-the-shelf software as much as possible, and as a direct result the system did not initially attempt to encode as data the interface rules, but instead hard- coded them using standard .Net development tools (form painters, etc. of the IDE). However, most other kinds of business rules were encoded as data this time. The resulting system required about 500,000 lines of C# code (excluding blanks and comments) and 160,000 lines of T-SQL code (including blanks and comments). It runs on a database server and terminal servers using MS SQL-Server and Windows Server 2003, supporting about 500 users having thin clients in the field. 3
- 5. The system was installed in September 2005. The development cost of the system was about USD $3.4 million, and an additional $1.1 million has since been spent on various fixes and upgrades. During this time the Company grew an additional 40%, to about USD $175 million. The monthly work hours dropped sharply after installation and continue to fall, as shown in Appendix A. There is still a team of five people performing a total of about 300-400 hours of work per month; this is expected to continue declining. Since installation the system has processed nearly seven million different “business transact- ions,” defined as (for example) a sale of one or more items, plus delivery, invoicing, cash application, and accounting and inventory entries as necessary. A transaction might also refer to a transfer of products between warehouses, purchases of products from manufacturers, or credit/debit memos and inventory adjustments. The Company’s work processes continue to evolve, for example (a) the high price of fuel necessitated delivery charges under certain conditions, and (b) the Company decided to charge customers a restocking fee for returns made more than thirty days after purchase. Analysis of Results The project took longer and was more costly than initially expected. An analysis of actual costs to date versus budget shows a 1.95: 1 ratio. According to Forrester Research, however, midsize wholesalers normally spend about 0.6% of gross revenues on software; using this metric, the actual cost was 89% of the industry average. Appendix B shows a chart of these costs. While the budget overrun was of course not good news, the comparatively low ongoing maintenance cost will provide a continuing savings that goes directly to the bottom line of the Company far into the future. The main drivers of increased project cost were as follows: Converting almost all initial data, including transactions in progress, from a non-relational to a relational database required a great deal of conversion coding; in addition many problems/errors were found in the source data that had to be resolved before installation. The problem was compounded because the vendor could not or would not provide documentation for its database schema. This problem was unique to this situation; it is not likely to be repeated, although any data conversion can be difficult. We have no special tools or theories to help with this. To meet the COTS requirement, we used standard Microsoft .Net tools, but found that a great deal of code – about five times what was expected -- was required to be written, primarily for the user interface. We are now moving to a design that still uses .Net but where many of these rules are represented as data, so that code can be eliminated and future changes to user interfaces will be far easier to manage. It would have been better to design this flexibility in from day one. When the project started to become overdue, the design process was curtailed and discussions were minimized “until after cutover.” This resulted in one major subsystem (cash application) being designed and implemented in a more complex manner than was really necessary if the 4
- 6. Company’s needs had been better understood. The subsystem then had to be redesigned and re- implemented with correct requirements after further discussions. Continuing to work on user requirements, when the problem but not its alternatives were known to exist, would have avoided this substantial effort. Lastly, by putting together a new team for this project, there was a steep learning curve in three areas: (a) the team needed to learn how to work together, which was painful, (b) the team needed to learn how to work in a geographically dispersed environment, with various team members all over the United States; and (c) the team needed to learn how to build and test a system where all rules were not easily found in code, but where the programmers and testers had to look in two places – the code and the rulebase – to understand what rules were in effect at any given time. It would have been better to have a team that had already worked together on similar projects, but that was not possible then. There is such a team now. Conclusions After moving most rules to data, the code that is left is mainly control logic that knows nothing about the world except what tables to look at, in what order, and what to do based on rules selected for execution. Since this control logic is unlikely to change over time, one benefit of this approach is that the software and data structures stay remarkably stable even as the rules continue to evolve. Another is that subject experts and business managers can explain new rules to business analysts (not only programmers), who can then directly update the rules through a RDBMS. In another prior project the subject experts directly updated the rulebase, without having to explain things first to a programmer or even an analyst. This eliminates the possibility of misunderstandings between subject-area experts and technical experts. Representing rules as data also means that the RDBMS must be optimized to work efficiently. This project has demonstrated the importance of minimizing the amount of software written and maintained. This can best be done by implementing both business rules and interface rules as data, not code. It has also shown the importance of readily changeable work process to facilitate the continuous evolution of business work processes. References Forrester Research, Inc (2005); US IT Spending Benchmarks for 2005. Long, J., and Denning, D. (1995); Ultra-Structure: A design theory for complex systems and processes; Communications of the ACM, Volume 38, Number 1 (pp. 103-120) Long, J. (1999); A new notation for representing business and other rules; Semiotica Special Issue: Notational Engineering, Volume 125-1/3 (guest ed. J. Long) (pp. 215-227) 5
- 7. Appendix A: Actual Hours Required 2,000.00 1,800.00 1,600.00 Cutover 1,400.00 1,200.00 1,000.00 800.00 600.00 400.00 200.00 0.00 Jun-03 Jun-04 Jun-05 Jun-06 Jun-07 Mar-03 Sep-03 Dec-03 Mar-04 Sep-04 Dec-04 Mar-05 Sep-05 Dec-05 Mar-06 Sep-06 Dec-06 Mar-07 Sep-07 Dec-07 Appendix B: Actual versus Ideal Costs and Industry-Standard Costs 200,000 180,000 Actual Industry Standard 160,000 140,000 Cutover 120,000 100,000 80,000 60,000 Ideal 40,000 20,000 0 Jun-03 Jun-04 Jun-05 Jun-06 Jun-07 Mar-03 Sep-03 Dec-03 Mar-04 Sep-04 Dec-04 Mar-05 Sep-05 Dec-05 Mar-06 Sep-06 Dec-06 Mar-07 Sep-07 Dec-07 6