Foresight Tips & Tricks

Take a look at the following article from the Washington Post: 

http://www.washingtonpost.com/wp-dyn/content/article/2008/03/31/AR2008033102789.html?wpisrc=newsletter&wpisrc=newsletter

Here’s a key quote from the conclusion:

Defense Department officials have tried to improve the procurement process, the GAO said, by doing more planning and review in the early stages of a contract. But “these significant policy changes have not yet translated into best practices on individual programs,” Gene L. Dodaro, acting comptroller general of the GAO, wrote in the report.

“Flagship acquisitions, as well as many other top priorities in each of the services, continue to cost significantly more, take longer to produce, and deliver less than was promised,” Dodaro said. “This is likely to continue until the overall environment for weapon system acquisitions changes.”

One of the keys to risk reduction in complex, technology-heavy systems development is up front analysis.  Everybody understands this and salutes it, but few really do it.  As mentioned in an early article, the reality is that a defense contractor is really incentivized to “do it wrong” a couple of times before doing it right.  This is at least somewhat due to the pressure to deliver demonstrate-able versions as quickly as possible, rushing the project through the critical requirements analysis and design phases.  There is also an instinctive understanding that getting the next round of funding is easier if I can quickly deliver something tangible, even if it isn’t designed right.  I can refine the design a number of times before delivery.

Unfortunately, this approach is extremely expensive in time and money, as any observer can easily witness.  Modeling & Simulation provide a powerful mitigator to this problem when used religiously from the requirements phase on through the project.  M & S not only improves the analysis process by flushing out the detail devils, but helps satisfy the strong engineering urge to have some tangible deliverable to interact with.  Essentially, a “virtual prototype” is produced that stakeholders can interact with in any number of ways to determine whether requirements are met, performance will be as expected, etc. 

Some will argue that models can be made to lie, and that is certainly true.  However, a model (and it’s embedded assumptions) is analyzable (by demonstration, test, and inspection) to determine whether it can be considered credible.  Contrast this with the hundreds of slides in presentation packages presented at design reviews which present just what the presenter wants the reviewer to see.

Sadly, it appears that simulatable models are only sporadically required as deliverables in government programs.  This is baffling considering how valuable such models can be in evaluating proposals and system designs.  Acquisition offices need to ensure that this valuable tool is written into RFPs and contracts for several reasons:

1. The modeling and simulation process will greatly assist in the analysis and design of the system, resulting in better system designs.
2. The resulting simulatable model will be extremely useful to the evaluators at various milestones.
3. The resulting model will be owned by the customer and can become part of a knowledge-base that can be useful to client programs, future increments of the same system, and the development of similar systems.

Modeling and simulation can be an expensive activity for complex systems, but don’t we have enough data to convince ourselves that not doing it is even more expensive?

We’ve been asked about Foresight’s application to JTRS enough times that we thought it best to just include the following here for reference. 

Early in the JTRS program, Foresight was contracted by JPO to develop a demonstration model for waveform/platform performance analysis and as a result, the use of Foresight was highly recommended to the GMR bidders by the selection committee.    

Over the last three years, Foresight has been used on several JTRS-related performance modeling projects at Boeing, Northrop and GDC, with some of the results being published in OMG workshops (see whitepapers).  This work has shown that Foresight-based performance modeling is effective and adds significant value to the JTRS program by providing: 

• Early radio design guidance and performance prediction including processor and memory sizing, technology selection, memory and processor utilization and power analysis.
• Waveform design guidance and performance prediction on different candidate platforms.  Performance metrics include startup time, packet latency, throughput, jitter, and QoS.
• Impact analysis of proposed requirement modifications, design changes, unmet constraints, or new waveforms.
• Performance models for use by JTRS client programs, such as FCS, that would be available before prototypes.
• Early detection and clear communication of component-level performance and timing/resource budget issues.

In addition, Foresight has been used on the FCS program for trade studies as part of the risk mitigation effort.  This effort used and built upon a performance model of the WNW waveform running on the increment 0 GMR platform developed earlier.  This work was highly successful and established the usefulness of a waveform/radio performance model to client programs.

Our experience has shown that, at the outset of performance modeling, only 50% of a model’s usefulness is envisioned.  Once that model becomes available, even if incomplete, many other uses of the model become apparent.  For instance, on the AMF program a performance model was used to perform a sensitivity analysis of the central non-volatile memory data read rate on startup time given a couple of different startup strategies.  This provided excellent information for component selection from a model that was originally intended to provide startup time and processor utilization.

Foresight’s approach is to build a model of the radio platform, a model of the waveform, and then execute the waveform model on the radio platform model.  The resulting composite model can accurately predict the behavior of the waveform when deployed on the radio.  With such a composite model, the radio performance can be predicted under many different environmental scenarios, traffic patterns, etc.  In addition, the model acts as an excellent experimental environment for trade studies to aid in design decisions or addressing problem areas.Moving from the radio model to systems-of-systems modeling at the network level (such as is required for FCS-type scenarios) can be accomplished in a number of ways.  The Foresight radio models are complete enough to be connected together for a multiple radio scenario but, for modeling larger scenarios, it is probably better to abstract the performance of the radio model into a node model within a network simulation or operational scenario simulation. 

In a nutshell, what we do is to create two models: (1) the radio hardware (CPUs, busses, memory, FPGA & DSPs) and (2) a model of the waveform, radio services and devices, operating environment (framework), CORBA, and RTOS.  Then we execute the model of the software on the model of the radio to evaluate how the software will perform on the radio platform.  We can model & measure:

• Processor utilization as a function of time
• Memory utilization
• Bus utilization as a function of time
• Packet or data latency & jitter through waveform
• Data throughput through the waveform
• Radio startup time
• CORBA overhead
• RTOS overhead

Using this model, we can also perform trade studies to determine optimal system configuration.  The same software model(s) can be executed on different radio models through a simple re-mapping activity in order to evaluate application performance on different radio platforms that you may have to build.

As always, we welcome your comments and questions to any of these BLOG articles.  However, it might not be obvious how to post a comment.  So, here’s some info…

In order to make it easier to comment on BLOG articles, we have eliminated the requirement that commenters be registered and logged in.  However, all comments are moderated in order to control spam. 

To comment, simply visit the post of your choice and add your comment using the box at the bottom of the page.  Name and E-mail address are optional.

One of the areas where I have been kept busiest as a Foresight modeling consultant is in creating models that predict processor utilization.  Frequently, system designers or customers want to design-in some CPU “head-room” capacity.  This may be motivated either by a desire to over-design slightly in order to ensure sufficient capacity in peak load situations and minor software revs, or a need to plan for future growth.  These requirements appear in a number of forms, but they all imply the need to predict and measure processor utilization.

In complex embedded systems with multi-team applications development, such as software-defined radio, “design to CPU utilization” can be very challenging.  One cannot simply allocate processor bandwidth to software units in the same way that one can allocate memory.  (Actually, you can, if the system is simple enough.  You can even enforce it like you can with memory.  An RTOS like Green Hills’ Integrity will let you create different “time” partitions.  Unfortunately, this methodology breaks down if the system is complex and has unpredictable loading behaviors.)  It becomes necessary to predict (and often to track) compliance with the CPU utilization requirement from the early stages of software design through test and signoff.

If you’re faced with a CPU utilization requirement, what are your options?  Here is a quick summary of a few methodologies that you should consider:

Read the rest of this entry »

For the last three years, I’ve spent the bulk of my time using Foresight to help companies perform performance analysis of their JTRS software defined radio designs.  In that time, I’ve noticed some things that I’d like to comment on.  At the risk of biting the hand that feeds me, so to speak, here are my observations and opinions (for what they’re worth.)

Read the rest of this entry »

We recently received the following question:

Do you have any examples of pulling real-time data from operations and using it to populate the simulation models so that they are continually re-initialized with current data?

Read the rest of this entry »

When we talk about performance modeling, we’re talking about system-level performance models.  The components in the system may be people, computers, trucks, networks, or electronics components.  The way Foresight works is that you use data flow diagrams (DFDs), state transition diagrams, and procedural “code” to describe the behavior of your system.  The diagrams are simple to understand and use, very much like you might draw on a whiteboard or the back of a napkin  This description can be executed, via discrete event simulation (DES), to explore it’s behavior. 

Read the rest of this entry »

Microsoft Excel can be a powerful allly when attempting to tame large Foresight modeling projects.  We have benefited greatly from its capabilities as applied to

• Parameter management
• Batch execution
• Data analysis

This article will deal with the first two, parameter management and batch execution.  In a future article, we’ll describe some powerful ways to use Excel and Access for data analysis.

Read the rest of this entry »

If you’re a Foresight user, and you’re actively modeling and/or debugging (as opposed to just doing batch simulations) we’d like you to seriously consider using the Chimera Beta.  This is the new Java NetBeans-based GUI that we’ve been working on for several years.  We’ve nicknamed it “Chimera” because it’s a little like that mythical animal (body of a lion, tail of a serpent, and goat head coming out of the back.)  We have taken the Foresight code-base and “remodeled the house while living in it”, making incremental changes to the GUI while leaving the simulator, data model, and database alone.  Read the rest of this entry »

I am often asked about the relationship or interaction between Foresight and some other modeling language, such as UML, SysML, or MatLab.  These are hard questions to answer in a sales situation.  The first thing the person is usually desparately hoping is that they can realize the performance simulation and analysis benefits that Foresight provides with a modeling language that they know and/or have existing models in.  Sadly, no modeling language is a Swiss Army Knife that can meet all of your specification and analysis needs (yet; we can still dream, more on this in a moment!) Read the rest of this entry »