Develop Top-Level System Design
Definition: In acquisition-oriented systems engineering, a top-level system design represents the envisioned implementation of a system in sufficient detail to support credible projections of cost, schedule, performance, evolution, and risk. A top-level system design can be used to assess system feasibility at the outset of a program, as a basis for performing analyses of alternatives, and as a tool to help finalize requirements and budgets prior to a system development contract solicitation. When developed with careful consideration of its purpose, the top-level system design becomes the program's early technical baseline for acquisition planning activities.
Keywords: cost analysis, cost analysis requirements document, early design, early system design, early systems engineering, requirements optimization, technical baseline, top-level system design
MITRE SE Roles and Expectations: During initial capability planning activities, the MITRE systems engineer (SE) is often involved in establishing a sound program baseline, which includes an understanding of the system operational requirements, the system design concept, the architecture, the technical requirements, and the associated program cost and schedule. In these situations, a MITRE SE is expected to:
- Understand the purpose and role of top-level system design in the acquisition process.
- Understand how and when a top-level system design should be undertaken.
- Understand the associated benefits and risks.
- Identify and engage subject matter experts (SMEs) with core technical skills appropriate for developing the top-level system design.
- Apply the top-level system design baseline and lessons learned from the design activity in program acquisition planning activities.
A top-level system design represents the government team's independent projection of the way a system could be implemented to meet the prevailing requirements with acceptable risk. Although a top-level system design could be mandated for eventual implementation by a development contractor, it is generally under the stewardship of the government team. The primary reason for developing a top-level system design is to provide a technical foundation for planning the program. It is the government's de facto technical approach to meeting the customer's needs. Once defined, a top-level system design represents an approach that can be used to develop a program schedule and cost estimates consistent with the program's technical content, as well as risk assessments, acquisition strategies, a logistics approach, etc. If a program is required to develop a Cost Analysis Requirements Description (CARD) , the top-level design can be the foundation of that work.
Because the government is not in the business of designing systems, the top-level system design is only representative of what can be built with conventional technology. Therefore, the design may sometimes indicate lower performance or greater cost than that of the design the contractor will eventually develop, which may include proprietary innovations (particularly in a competitive environment). MITRE SEs are expected to help lead the government effort to create realistic top-level designs and associated risk mitigation activities so that planning will be based on a realistic foundation. Cost, schedule, and performance projections based on the top-level system design should include margins that will tend to help mitigate program risks.
As with many aspects of acquisition-oriented systems engineering, the role that the top-level system design plays varies considerably across programs. In some programs, any design—even a "top-level" one—is viewed as treading excessively into implementation territory and is therefore considered the responsibility of the development contractor rather than the government. Even programs that do not share this philosophical view often do not have enough systems engineering resources to develop a top-level system design and the mandated architectural frameworks. The effort needs to be a teaming approach of the government, MITRE SEs, support or development contractors, and industry. In some cases, academia will also have a role in helping with ideas for technology and capabilities that could be included in a top-level design.
The process of developing a top-level system design engages MITRE SEs in the program technical content and provides numerous benefits:
- By analyzing the known/draft system operational requirements, the SE can discover the design implications of these requirements, identify any potential requirements conflicts or technically infeasible requirements, review and comment on the operational documents being developed, identify and possibly initiate requirements trades and/or risk reduction activities such as prototypes and experiments, and assess the affordability of the required system. This allows for smarter early interaction with the authors of the operational requirements document(s), producing clearer expectations and better defined operational documents (see the SEG's Requirements Engineering topic).
- By developing a top-level system design, the SE discovers the complexities, dependencies, and interactions within the system, gaining a better understanding of the program's technical concerns, issues, evolution needs, and risks that will have to be managed during program acquisition. The natural interplay between a top-level system design and system architecture should be captured in each (see the SEG's System Architecture topic).
- Developing the top-level system design requires the SE to explore industry's (and, at times, academia's) capabilities and the available technologies. This includes assessing the interest and capability of potential contractors and the maturity and availability of required technologies. Early involvement with the potential contractors for the program tells the SE what questions to ask in the request for information (RFI) solicitation, enables intelligent discussions with industry during Industry Days, and results in a clearer request for proposal (RFP) package, resulting in better quality proposals. A good understanding of the maturity (or lack thereof) of relevant technologies helps the SE define appropriate program acquisition and risk mitigation strategies. Communicating the users' needs and requirements together with the top-level design to industry partners is key to helping them understand the need and formulating the best solutions for the capabilities.
- Developing the top-level system design requires the SE to investigate availability, maturity, and applicability of products (systems, subsystems, components, algorithms, etc.) that could be used to provide some of the required system functionality and performance, thus getting an early assessment of the risk/benefit trade-offs associated with these products.
- The top-level system design process reviews the technical content and lessons learned from any precursor programs. The deployed capabilities, technologies, products, approaches, and solutions are available to the SE to consider for the new program.
- Developing a top-level system design essentially mimics the first part of a system design and development program, stepping through abbreviated phases of requirements definition, decomposition, analysis, assessment of alternative solutions, and development of a design.
A top-level system design may be independent of technology, contractors, and existing systems. It needs to be detailed only enough to provide inputs into program cost and schedule estimation activities to allow for program planning activities (acquisition strategy, risk reduction, etc.). However, with many interfacing capability needs, use of services from repositories, and constraints of the technical environment where the capability must reside, a design cannot be completely agnostic to the technology and existing systems and must consider the overall context and enterprise into which the capability will reside (see the SEG's Enterprise Engineering section).
Finally, top-level system designing is a recurring activity. Initially, it can start as a refinement of the alternative selected from the analysis of alternatives; in time, it is updated and fleshed out as more information becomes available after the technology readiness assessment, RFI/Industry Days, user requirements working groups, completion of risk reduction activities, etc.
The Top-Level System Design Process
Figure 1 shows how the top-level system design serves as the core of the program office's systems engineering analysis loop. It represents a snapshot taken just prior to issuance of the RFP for the system development phase, at which point the major systems engineering challenge facing the program was the finalization of system requirements and the program budget. However, the same process was used earlier in the program to establish system feasibility and to identify and assess the maturity of critical technologies. Subsequently, after system acceptance, planning and budgeting for upgrades can be established.
Best Practices and Lessons Learned
Representative design. During the top-level system design activities, the SE has to be continually aware that the top-level system design may be just one of many possible solutions. The SE should keep an open mind to all available designs and ensure as "generic" a top-level system design as possible, one that will enable competition by all relevant industry members. Absent compelling performance reasons, the top-level system design solution should avoid locking in on a one-party approach (e.g., using an idea, solution, or existing product available to only some contractors).
Competing designs. Sometimes two mutually exclusive approaches are possible for the system design (e.g., software intensive vs. hardware solutions, or reuse of extensive existing products vs. new development). In this case, basing the program plans on a single top-level system design may have negative consequences because it could result in the development of system requirements that would preclude the bid of one (or more) contractors or result in unrealistic program cost/schedule estimates. In this instance, the SE should help evaluate the alternatives, identify and consider their pros and cons, and carefully decide which top-level system designs should be developed and carried forward into program planning to cover the range of implementation options.
Applicable, feasible, affordable, phased. Another challenge for the SE is to keep the top-level system design feasible and affordable and not let it fall into the trap of capturing the best features and all the additional capabilities of the various possible approaches, alternatives, technologies, and products. It is important to ensure that the top-level system design meets the stated operational needs and is affordable and available in useful time. The implementation of solutions to complex problems is likely to be time-phased. Therefore, the top-level design should plan for evolutions of capabilities based on urgent user needs, technical feasibility, and affordability over time. Additional "bells and whistles" may be exciting, but they should be avoided because in the long run they risk breaking the program's bank.
Design depth. Developing a top-level system design is hard for SEs. By definition, the top-level system design is not intended to resolve all problems, address all issues, or mitigate all technical risk. Letting go of a partially completed design can be hard. The trick is to understand which parts of the top-level design need what degree of depth in order to address the critical uncertainties to be explored at this stage of the life cycle. That depends on an assessment of the key risks for the particular program and the relevant technologies.
An example of a comprehensive top-level system design comes from a government acquisition program that developed its capability within budget and ahead of schedule, earning the government-contractor program team accolades for exemplary acquisition excellence. MITRE served as the lead systems engineering organization for the government program office team, which included multiple systems engineering and technical assistance contractors, government laboratories, and other federally funded research and development centers.
One of the cornerstones of the program's systems engineering approach is the conviction that the best way for program offices to obtain reliable projections of cost, schedule, performance, and risk is through independent analysis. A comprehensive top-level system design, together with a government team with enough subject matter expertise to develop and use it, is the crucial enabler for such analysis. The following lessons learned are based on MITRE’s experience with the government acquisition program.
Depth is key. To support meaningful assessments of feasibility, cost, and performance, a top-level system design must be detailed enough to identify and describe critical items. The level of detail needed to get to "critical items" depends on enabling technology, requirements, and the like. In this particular program, sufficient detail required identification and description of critical items such as enabling chips and components (e.g., low-noise amplifier monolithic microwave integrated circuits, high-power-density direct-current to direct-current converters, heat exchangers), key algorithms (e.g., false target mitigation), and all required computer software configuration items. The required depth was about the same as that of the technical volume of a system development proposal; in the case of the referenced program, the top-level system design document was more than 300 pages and extended to work breakdown structure (WBS) Level 4—and in some cases, Level 5.
A typical objection against this degree of depth is that "nobody is ever going to build this government design anyway, so it's better to base programmatic projections on conceptual designs that competing contractors provide." One answer to this objection is that costs and risks are largely determined by decisions made early in a program's life, well before any contractor-developed conceptual designs exist and often before candidate contractors have been identified. A more compelling answer is that, in a competitive environment, there are strong incentives for even the best-intentioned contractors to underestimate the challenges associated with meeting the requirements. Conversely, in a sole-source environment, there may be strong incentives for the contractor—particularly one who is already producing a system that could be viewed as a competitor to the envisioned system—to overstate those challenges.
For example, in the referenced program, the original operational requirements called for capabilities that in many ways exceeded that of any extant system. At that time, other systems performing a similar mission were not only less capable but also had to be tightly integrated into their host platforms, requiring extensive platform modifications. In contrast, to minimize cost and maximize operational availability, the operational user organization wanted the bulk of the system mission equipment to be encapsulated in a self-contained module that could be rapidly moved between smaller—and more lightly modified—host platforms. At the time these requirements were floated, the industry consensus was that they were infeasible. However, the comprehensiveness and depth of the top-level system design provided the sponsor with the confidence to properly budget and proceed with the acquisition, which eventually proved highly successful.
Don't skimp on the team. The team required to develop a sound top-level system design of the requisite technical depth is substantial—about 30 to 40 people in the case of the referenced program—with total effort of about 30 staff-years. Of these, about 70 percent were SMEs, including hardware and software engineers, cost analysts, operational analysts, and experts in modeling and simulation. Approximately 20 percent were process experts, responsible for planning, configuration management, requirements traceability, and acquisition process compliance. Finally, 10 percent were technical leaders, including a chief engineer and system architect. Critical to all of this is involving the end-user community in the design process all along the way to help with trade-off analysis and negotiation of needs.
For this particular program, the same demographic makeup was also about right for virtually all of the other technically intensive government systems engineering activities necessary to support this acquisition, including proposal evaluation. For example, the top-level system design team eventually formed the core of the cost and technical boards that advised during proposal evaluations; their experience in developing and costing the top-level system design was invaluable in ensuring thorough proposal evaluations.
Involve cost analysts in the design synthesis. A key use for a top-level system design is to serve as the basis for cost projections; a properly documented top-level system design meets all the requirements for the DoD-mandated CARD .
The referenced program's experience shows there is substantial room for error in the subsequent cost estimate, no matter how thoroughly a design is documented, unless the cost analysts are involved in the design synthesis. Cost analysts can substantively contribute to the design effort by helping to define an appropriate WBS, which effectively serves as the outline for the top-level system design document, and by identifying the type of technical information that should be documented for each WBS area to facilitate costing.
An added benefit of involving the cost analysts up front is that cost models can be developed in parallel with the design synthesis, enabling the cost estimates to be provided sooner (about two months sooner, in the case of the referenced program) than those developed in a sequential process.
As previously mentioned, a crucial benefit to participating in the design process is that it gives the cost analysts greater insight into the salient technical issues, enabling them to thoroughly evaluate the cost proposals the contractors submit. Although difficult to quantify, this benefit was evident in the program's source selection.
MITRE's organic cost analysis capability and experience in tightly integrating that capability with extensive in-house technical subject matter expertise was unique among the many organizations supporting the acquisition program, and the resulting synergy was of significant added value to the program.
Involve performance modelers in the design synthesis. The top-level system design serves as the basis for system performance projections. As with cost estimation, involving operational analysts and performance modelers up front allows the required models to be developed sooner, while improving the fidelity of the results.
Early development of the performance model provided a key benefit for the referenced program: it enabled the government to include the MITRE-developed model—but without the design parameters specific to the government design—as government-furnished information in the RFP packages provided to candidate offerors. Offerors were required to provide design-specific input files for the model as part of their proposals. This enabled the government to use a common tool to rapidly evaluate the performance of the competing designs—the same tool that was used to help set the requirements in the first place.
This continued to pay dividends beyond the source selection. The program's development contract called for the selected offeror to include a performance model as part of the delivered system (e.g., to support mission planning). The contractor was free to adapt an existing model for this purpose or to develop one from scratch. Because the MITRE/government model had been developed and already had been validated within the government team, the contractor elected to use it as the basis for the delivered model, thereby reducing cost and risk.
Organizationally collocate the key members of the team. Developing a sound top-level system design requires extensive face-to-face communication, necessitating that most team members be physically collocated. However, even with physical collocation, risks increase when the team is spread across multiple organizations and multiple management chains of command. The experience of the referenced program shows that synthesizing and leveraging a top-level system design becomes problematic unless key hardware SMEs, key software SMEs, and key cost analysts are all resident within the lead systems engineering organization. Although detailed-level design and implementation activities may sometimes be well accomplished by allocating tasks to hardware, software, and cost support organizations that are "best" in each discipline, early design work should be accomplished by a close-knit team representing all the required specialties, as well as overall systems engineering leadership.
Ensure the top-level system design is an independent government product. Contractor-developed conceptual designs may already exist when the government team undertakes development or update of a top-level system design. Because the top-level system design is used to help set requirements and budgets, the inclusion of any of the contractors' proprietary innovations in the government design—thereby revealing an ostensibly preferred design approach—could destroy the integrity of a source selection. The government and MITRE must be cautious stewards of the top-level design. If the design is shared with industry, it must be shared with all interested industry partners equally. Even if care is taken to avoid the inclusion of proprietary features, however, a top-level system design often resembles at least one of the contractor-developed conceptual designs. This should present no problems, provided that a paper trail can show the design was independently derived. For example, the referenced program's top-level system design provided an extensive design rationale that illustrated how the design flowed naturally from the requirements, based on technologies and sensor design practices documented in the open literature.
References and Resources
- Defense Acquisition University, Cost Analysis Requirements Description (CARD), Acquipedia, accessed July 29, 2015.