1
Projecting Future Electronic Storage Requirements Within
the Client/Server
A Report Prepared for
NC DOT
by
William Rasdorf
2
1. Problem Statement
As technology advances, the need to store data in electronic format increases. Often, it is difficult to project server hard drive storage requirements and, if an adequate storage plan is not in place, servers have a shorter lifetime than that which is anticipated when they are purchased and installed. This report will discuss issues related to project server disk space and will discuss the items in the following outline.
2. Outline
1. Problem Statement
2. Outline
3. Case Study Storage Requirements Case Study Scenario Electronic Storage
Text Drawings Photographs Project Summary Other Files Applications Total Summary Archiving
Paper Storage
4. Future Electronic Storage Issues Access to Information Managing Information Software Archives
Compatibility Requirements Hardware Requirements and Costs Software Requirements and Costs
5. Traffic Surveys Case Study
6. Plan
Work Items Level of Effort
7. Summary and Conclusions
3
3. Case Study Storage Requirements
In order to provide an understandable frame of reference for such a task as undertaken here - that of projecting future electronic storage requirements - a hypothetical consulting structural engineering firm will be used. Prior to his current position the author was a consulting structural engineer in the Washington DC area for over 3 years. Although this case study requires a conceptual translation for the NCDOT it establishes a firm foundation that is rooted in specifics and details. Therefore, it is deemed to be useful and relevant.
3.1. Case Study Scenario
The size of the firm is approximately 50 employees with a projected workload for the current fiscal year of 120 projects of varying size. These may range from residential inspections to multistory office design. Currently all projects utilize computers for the creation of documents from drawings to correspondence. However, many completed projects (especially those completed over 10 years ago) have no electronic form of record. The problem of storing an electronic library of project information must be confronted in a methodological manner. An average project is assessed to determine projections for hard disk space for the entire office.
3.2. Electronic Storage
One approach within an office for the storage of project information is to store each project of the network server in a "projects" directory under a sub directory named by the project number. To further facilitate the organization of the project and its storage, more sub directories are created with headings like the following: Project management, correspondence, faxes, transmittals, minutes, drawings, specifications, construction administration, and miscellaneous.
p:\projects 96005
files 96434
files 97400
01pm 02corr 03fax 04trans 05min 06draw
4
Each of these sub directories may again be subdivided on a project-by-project basis as additional items might need to be included. In a windows operating system, the file manages may show a directory organization as shown below.
Of course, each of the sub directories shown contains a myriad of electronic files. Most of the files are output as 8 1/2" x 11" pages, but some (specifically those in the 06draw sub directory) can be rather large drawing files for D and E size engineering drawings. Each of the sub directories will be analyzed according to an assumed average project size and a total duration of 30 months from conception through construction.
3.2.1. Text
Most of the digital files in a project directory are text files or spreadsheets which have minimum storage requirements. Let us assume that the average correspondence, fax, transmittal, or meeting minutes requires 50K (kilobytes) of hard drive space. Although the density of documentation varies throughout the project, the transmittal sub directory (04) may contain as many as 80 files. This number will be used as an average for each of the eight (8) non-drawing related sub directories. Therefore, the total number of text files can reach 700 (640) or 35MB (megabytes of storage) by the end of the project.
3.2.2. Drawings
Drawing files are normally much larger than text files. A typical AutoCAD file will contain 2M of information. If an average project requires a series of 20 structural drawings, storage requirements can reach 40M.
3.2.3. Photographs
During construction administration, it is becoming standard practice to use a digital camera for recording site information. The project engineer will typically obtain 20 photographs during a weekly site visit, so a construction time of 50 weeks may yield upwards of 1000 photographs. A digital camera with low resolution can typically store 20 photographs in 4M of disk space, so an entire project can be stored in approximately 200M on the network server.
3.2.4. Project Summary
5
storage capacity will be required for 120 projects that span the fiscal year. Five (5) years of projects will, therefore, require 165G of storage capacity.
3.2.5. Other Files
In addition to a project directory other major network directories are necessary to store marketing files, personnel and payroll files, and miscellaneous files. The marketing directory will probably contain hundreds of scanned images which can each be as large as 5M. Assume that the marketing directory contains 2G of information, that the personnel and payroll directory contains 10G of information (200M for each of the 50 employees) and the miscellaneous directory contains 2.5G of information such as standard forms, resumes, telephone and address directories, standard design specification templates, and other items.
3.2.6. Applications
Application files in a structural engineering office are relatively small when compared to the overall project files. Applications of a primary nature would include Word, Excel, AutoCAD or MicroStation, ANSYS, Netscape, RamSteel, PhotoShop, and payroll and business applications. For these 8 primary applications we estimate approximately 50M for each for a total of on the order of 500M for applications.
3.2.7 Total Summary
To summarize all of the projects for 5 years and estimate total resource requirements we have the following table.
directory storage requirements
projects 165G
marketing 2G
personnel 10G
miscellaneous 2.5G
applications .5G
total 180G
3.3. Archiving
6
On-Line network storage gives all workstation users direct access to network hard drives which contain all project information for current projects. Access is instantaneous.
Off-Line storage used CD-ROMs to store all project information for completed projects. Access is more difficult and timely, but a well organized filing system will allow an information systems manager to easily locate any file within a few hours.
Near-Line storage utilizes optical discs that store any file that has not been modified for a certain length of time ( a time stamp is necessary for this to operate effectively). Each workstation has access to a CE "junk box" where optically archived files can be accessed relatively quickly (within several seconds, as opposed to immediately) and copied again to on-line network drives.
3.4. Paper Storage
Up until about 10 years ago, most projects were drawn by hand and correspondence (although created digitally) was stored in standard five drawer file cabinets. Typically a single project would require two full drawers of space and a full year of projects could approach 48 five drawer cabinets for 8 1/2" x 11" documents. A record drawing set would also be stored and a full year of projects would require 6 ten drawer flat file cabinets.
It should be noted here that paper files will continue to be required until government regulations are changed so that electronic documents can be more readily accepted in court to settle disputes. Additionally there may be many other reasons why paper files will continue to exist, paramount of them all is the cost of converting them to digital format.
Finally, if an engineering firm, or a government organization has been operating for more than 20 years, most archived paper files are either stored off site or in large libraries that are not readily accessible. Additionally, some of them may have been converted to microfilm.
4. Future Electronic Storage Issues
4.1. Access to Information
7
previous projects as a database for current ones. Additionally, new employees may find it useful to browse recent projects to become aware of the firm's methods and procedures and the documents they use and produce. Information is a resource almost equal in value to the employees themselves.
When speaking of access to information the following need to be considered: time of access, reliability, and management. A brief discussion of access time and reliability is appropriate here. While there exist many types of data storage media, they vary greatly in terms of the ability to readily access information and their reliability and life span. On line, off line, and near line storage have been discussed previously and should be considered when access is concerned. In addition some types of media are not suited for ready access. These include 1/4" and 1/2" magnetic tapes and 4mm and 8mm DATs. Other types of storage media are well suited for more immediate access to information. These include floppy disks, optical disks, and CD-ROMs. In terms of reliability, it is widely understood that the life span of a magnetic tape is approximately five years while the life span of a CD-ROM can be well over 100 years, provided that the CD itself is well treated.
4.2. Managing Information
It is apparent that the access of data will require an organizational system that will not only ease, but limit access to appropriate personnel. Computer files are usually stored in a hierarchy of folders of directories which have the benefit of imitating what many offices currently do with paper files. Therefore, locating certain electronic files becomes a simple translation from the paper world to the digital world. As data is archived, a directory tree structure similar to that discussed in Section 1.1 can be implemented. Another option is to have employees purge project directories before archiving in order to eliminate duplicate data and establish a more efficient filing system. This second option, however, is more costly and inefficient and can be eliminated by using identical filing systems and strategies while a project is active and as it is being archived.
8
outside via the world wide web on a project by project basis, or for use by different units, further management will be imperative.
4.3. Software Archives
One critical item that must be considered is the archiving of applications software. A careful organization wide plan should be developed for archiving the most commonly used software like MicroStation CAD software.
On a consistent basis this type of software is upgraded and changed and new versions are issued. It is critical that old versions not be discarded under the assumption that the new versions can always run using older data files. Rather, older data files must be carefully tagged so that they indicate the software version for which they are suited and that version of the entire software application should be archived.
4.4. Compatibility Requirements
As software and hardware requirements change over time, a base standard for reading and writing data needs to be sued. For example, such standards are in place for word processing (ASCII) and drafting systems (IGES). As computer technology becomes more advanced, such standards need to remain in place in order to be able to retrieve important digital information in the future. Standards committees and organizations (such as the American National Standards Institute - ANSI) are currently in place to address these issues. As the organization invests potentially millions of dollars in new software, compatibility issues need to be considered.
4.5. Hardware Requirements and Costs
Let us now look at the requirements and costs of storing both current project data and digitally archiving data for projects that have been completed. Although costs were not a consideration in developing a methodology they are useful to note. Look first at hardware costs for storage media. If the organization decides to electronically store existing paper files there are several options.
1. Scan small (text) documents on a flat bed scanner to convert them to image files.
2. Scan small (text) documents using optical character recognition technology to convert them to text files, thus saving valuable hard disk space.
3. Scan large (drawing) documents using large format scanners.
9
Assume that the organization decides to purchase scanning hardware for small documents. A scanner and software utilizing optical character recognition may cost $1000. For large format drawings, an outside CAD scanning and conversion firm will be employed. The conversion equipment used to scan the drawings will need to be able to recognize text, layers, geometry, symbols, and dimension styles.
Drawings can typically be scanned and converted to a recognized drafting format for approximately $50 per sheet. If the organization has been in operation for 20 years, upwards of 500 projects could need to be archived. Assuming again that each project contains 20 structural drawing sheets the total may approach 10,000 sheets, each requiring 2M of storage space for a total of 20G (drawings) and costing $500,000.
As text files are converted using optical character recognition technology, valuable storage space is saved. However, a 500 project database of text files will still require 17.5G of storage space (text) based on assumptions outlined in Section 1.1.
Therefore, to adequately archive all small and large documents using an optical character recognition scanner and an outside CAD conversion firm will produce 37.5G of storage capacity requirements (unless data compression is used) and cost on the order of $500,000. This large sum of money does not include the time required to scan the small documents. The value of converting old materials to a digital format should, therefore, be weighed heavily against its cost.
The cost of the storage media itself is a consideration. The attachments detail costs on the order of those shown here and are obtained from literature from Luminex Enlightened Storage Solutions.
media cost
CD-ROM $5 each
CD writer $500
DAT tape drive $600
CD tower with server $1500
10
Note that these cost figures would need to be verified for any specific application of the methodology and analysis. They are used here to give reasonable approximations of the costs involved so that their order of magnitude can sensibly be determined.
4.6. Software Requirements and Costs
In addition to the hardware and media costs, software will need to be acquired to organize, input, store, manage, access, and monitor the increasing collection of data. Most often, software is included in the purchase price of such items as scanners, digital cameras, CD writers, and the like and will not be an issue. If the software does need to be purchased, the price will probably be low enough as to not have much impact on the total cost of storing large amounts of information It is more likely that hardware and media costs will be the most important factor when deciding to archive project information.
Technical support will also be necessary in a computer world that is becoming increasingly sophisticated and specialized. Again, most technical support can be purchased for a nominal yearly fee, or (as in the case of most medium and large engineering firms) a designated person or department will be employed for the sole purpose of managing the information system of the entire office. This is not to suggest that such a strategy is ideally suited for the NCDOT. Rather, it is merely a consideration given the presence of the Engineering Automation Branch.
5. Traffic Surveys Case Study
A case study of the Traffic Surveys Unit of the NCDOT is proposed. The case study would consider all of the items noted in this report and it would include items as categorized below.
Present computer use
analyze existing server
analyze existing personal computers other
Determine document types, size, composition text
drawings raw data applications other
11
Questions
what can be digitally stored? what should be digitally stored? frequency of access?
access by who?
read, copy, and/or write access?
does data require an application to process it? other
The reader is cautioned that this list is by no means complete. Rather, the full and formal expansion of this list comprises work item 5 presented in the next section. The items shown here provide an indication of the direction in which the work is heading.
6. Plan
6.1. Work Items
As presently envisioned this plan consists of an overall sketch of activities, some of which have been done. Others are herein proposed. The items are as follows.
1. A number of professional resources (literature search) were investigated with a consideration of traditional literature (text and articles) but with a significant utilization of web sites.
2. Engineering firm case study and report to DOT. The goal is to realistically consider, formalize, and document the scope of issues involved even though the application domain is a fictional engineering firm rather than a DOT unit. Considered in the case study were text, drawings, photographs, miscellaneous documents, and application software. The focus of the case study was primarily on design and design documents.
3. Develop the plan.
4. Conduct another case study and report, this one specifically for the Traffic Surveys Unit of the NCDOT. This will consist of a detailed repeat of step 2 above. It will enable the identification of the items, issues, questions, considerations in the context of a DOT unit and will support the expansion of the items to be considered and allow further definition of the procedure needed for proceeding with other units.
5. Refine the plan.
6. Develop a survey for different units which can be used to gather the data they would provide to perform such assessments. Questions would be directed to senior technical managers and to the hands on users.
7. Work with Engineering Automation to identify NCDOT purchasing trends, to validate some of the quantification estimates, and to obtain their projections for software type and use.
8. Apply, refine, and generalize the survey and apply it to a data collection study for one additional unit. Meet with technical managers and users and assist them in the completion of the survey.
12
10. Develop a planning document for the units that includes the survey and information about and procedures for: how to collect the data; how to conduct a self assessment to determine needs; how to develop a plan to migrate to a hardware system to meet the needs; and how to implement the plan.
6.2. Level Of Effort
It should be noted that this report constitutes a summary of the work done under phase 1, consisting of steps 1, 2, and 3 above, and requiring 40 hours of time. Phase 2, consisting of steps 4, 5, and 6 is next proposed, permission is sought to proceed with these items, and their estimated time is 60 hours. A phase 3 consisting of steps 7, 8, 9, and 10 is also proposed ,and should be considered as a potential research project for the next fiscal year. To do a proper and comprehensive job on Phase 3 will require a substantial effort.
It should be noted that a great deal of knowledge can be gained by undertaking phase 2 even if phase 3 could not be funded as a research project. Much useful information would be gained that could be used by DOT units and clearly, the information gained by the Traffic Surveys case study would be valuable.
7. Summary and Conclusions
The intent of this report is to provide a broad overview of the possibilities of storing large amounts of information electronically. A typical engineering firm was used to provide an initial test case to study the process and the issues involved, however, the applicability of the process and information outlined here apply for NCDOT as well. Many of the assumptions made were conservative in order to allow for future growth and miscellaneous unanticipated needs that will constantly arise.
It has been shown that the cost of converting large amounts of existing paper (text and drawings) is substantial and probably not cost effective in most cases. There may be some instances or situations, however, that will require digital backup and this need can be assessed on a case by case basis as the need arises.
13
Finally, significant consideration must be given to the archiving of application software and even the hardware platforms on which it operates. Included among this strategy is the archiving of the documentation for the operation of the software and hardware. Too often software may be archived and save but it cannot be run at the future time that it is needed.
It must be emphasized that any digital data storage strategy must be designed. It must begin with a information collection effort (survey). It must include an analysis of the results of that effort. It must develop an organizational unit plan and implementation strategy. It must assign financial and labor resources to carry out the plan. Finally, it must continue to utilize the procedures and practices set up in the plan over a long term.
A large amount of information exists on the subject matter of data storage, especially as it relates to the special needs of the engineering community In addition, most trade magazines contain articles and advertisements relating to this effort in each issue. It is a widely discussed topic and one that each organization should assess seriously.
