Command center processing and display system replacement (ccpds-r) - Case Study

Command Center Processing and Display system- Replacement (CCPDS-R) Case Study

1. CASE STUDIES
•CommandCenterProcessingandDisplaysystem-Replacement(CCPDS-R)

2. 1. CCPDS-R Case Study
•CommandCenterProcessingandDisplaySystem–
Replacement
•TRWSpaceandDefenseinRedondoBeach,CA
•Customer:U.S.AirForce
•Focus:CommonSubsystem
•Missioncriticalsoftware

3. 2. Common Subsystem
•Primarymissilewarningsystem
•Maininstallation:CheyenneMountain
•Backupsystem:OffuttAirForceBase,Nebraska
•48-monthsoftwaredevelopmentschedule
•355,000SLOC
•Ada:design&implementationlanguage
•6buildswererequired
•Completedsuccessfully,ontimewithinbudgettocustomer
satisfaction

4. CCPDS-R Acquisition
•Conceptdefinition(CD),12-monthschedule
•5majorbidders,2contractsawarded
•Full-scaledevelopment(FSD)
•1contractawardedtoTRW
•ContractawardbasedonperformanceinSoftware
EngineeringExercise.
•Thewinningcontractorsalsoinvestedtheirown
discretionaryresourcestodiscriminatethemselves
withthebest-valueFSDphaseproposal.

5. CCPDS-R life-cycle overview

6. Concept Definition (CD)
•Vision
•Analyzeandspecifytheprojectrequirements
•Defineanddevelopthetop-levelarchitecture
•PlanFSDphasesoftwaredevelopmentactivities
•Configuretheprocessanddevelopmentenvironment
•Establishtrustandwin-winrelationshipsamongthe
stakeholders
•Softwareengineeringexercise

7. Process Overview for FSD
•Standard DOD life cycle after contract award
•Software requirements review (SRR)
•Interim preliminary design review (IPDR)
•Preliminary design review (PDR)
•Critical design review (CDR)
•Final qualification test (FQT)

8. 3. PROJECT ORGANIZATION
CDPhaseTeamResponsibilities:
•Analyzeandspecifytheprojectrequirements
•Defineanddevelopthetop-levelarchitecture
•PlantheFSDphasesoftwaredevelopmentactivities
•Configuretheprocessanddevelopmentenvironment
•Establishtrustandwin-winrelationshipsamongthe
stakeholders

9. 4. COMMON SUBSYSTEM PRODUCT OVERVIEW
IthassixComputerSoftwareConfigurationItems(CSCIs):
•NetworkArchitectureServices(NAS).Itprovidesreusablecomponents
fornetworkmanagement,interprocesscommunications,initialization,
reconfiguration,anomalymanagement,andinstrumentationofsoftware
health,performance,andstate.
•SystemServices(SSV)-ThisCSCIcomprisedthesoftwarearchitecture
skeleton,real-timedatadistribution,globaldatatypes,andthecomputer
systemoperatorinterface.
•DisplayCoordination(DCO).ThisCSCIcompriseduserinterface
control,displayformats,anddisplaypopulation.
•TestandSimulation(TAS)–Itcomprisedtestscenariogeneration,test
messageinjection,datarecording,andscenarioplayback.
•CommonMissionProcessing(CMP)–Itcomprisedthemissilewarning
algorithmsforradar,nucleardetonation,andsatelliteearlywarning
messages.
•CommonCommunications(CCO)–Itcomprisedexternalinterfaceswith
othersystemsandmessageinput,output,andprotocolmanagement.

10. Distinguishing characteristics of each CSCI

11. Full-scale development phase project organization

12. Software Architecture Skeleton
•AllAdamainprograms
•AllAdatasksandtaskattributes
•Allsockets(asynchronoustask-to-taskcommunications),socketattributes,and
connectionstoothersockets
•Datatypesforobjectspassedacrosssockets
•NAScomponentsforinitialization,statemanagementofprocessesandtasks,
interprocesscommunications,faulthandling,healthandperformancemonitoring,
instrumentation,networkmanagement,logging,andnetworkcontrol.
Figure shows perspectives of Software Architecture Stability (SAS)

13. 5. Process Overview
•TheSRRdemonstration:initialfeasibilityofthefoundationcomponents,and
basicusecasesofinitializationandinterprocesscommunications
•TheIPDRdemonstration:thefeasibilityofthearchitecturalinfrastructureunder
theriskiestusecases,includingthefollowing:
•ApeakdataloadmissilewarningscenarioofamassraidfromtheSoviet
Union
•Apeakcontrolloadscenarioofasystemfailoverandrecoveryfromthe
primarythreadofprocessingtoabackupthreadofprocessingwithnolossof
data
•ThePDRdemonstration:adequateachievementofthepeakloadscenariosandthe
otherprimaryusecaseswithinafull-scalearchitecturalinfrastructure,includingthe
othercritical-threadcomponents.
•The overall CCPDS-R software process had a well-defined macroprocesssimilar to
the life-cycle phases described in Figure.

14. Figure shows CCPDS
-
R
macroprocess
, milestones, and schedule

15. 5.1 RISK MANAGEMENT: BUILD CONTENT
•Build0-Itcomprisedthefoundationcomponentsnecessarytobuildasoftware
architectureskeleton.Theintertask,interprocesscommunications,generictaskand
processexecutives,andcommonerrorreportingcomponentswereincluded.
•Itistheconclusionoftheresearchanddevelopmentprojectexecutedinparallel
withtheCD(inception)phase.TheseNAScomponentswerethecornerstoneofthe
architecturalframeworkandwerebuilttobereusableacrossallthreeCCPDS-R
subsystems.Theyrepresentedverycomplex,high-riskcomponentswithstringent
performance,reliability,andreusabilitydemands.
•Build1.Thisbuildwasessentiallythe"architecture."Itincludedacompletesetof
instantiatedtasks(300),processes(70),interconnections(1,000),states,andstate
transitionsforthestructuralsolutionoftheCCPDS-Rsoftwarearchitecture.To
achieveacyclingarchitecture,thisbuildalsoaddedalltheNAScomponentsfor
initialization,statemanagement(reconfiguration),andinstrumentation.Atrivial
userinterfaceandthecapabilitytoinjecttestscenariosintothearchitecturewere
addedtosupporttheinitialdemonstration.
•Build2.Thiswasthefirstbuildofmission-criticalcomponentsandachievedthe
initialcapabilitytoexecuterealmissionscenarios.Thethreeprimaryrisksinherent
inthemissionscenarioswerethetimelinessofthedisplaydatabasedistribution,the
performance(resourceconsumptionandaccuracy)ofthemissilewarningradar
algorithms,andtheperformanceoftheuserinterfaceforseveralcomplexdisplays.

16. Common subsystem builds

17. 5.1 RISK MANAGEMENT: BUILD CONTENT
•Build3-Thisbuildcontainedthelargestvolumeofcode,includingdisplayformat
definitions,globaltypedefinitions,andrepresentationspecificationsneededfor
validationofexternalinterfacetransactions.Althoughthecodewasvoluminous,
muchofitwasproducedautomaticallyinacookbookmannerbyconstructingcode
generationtools.
•Theremainingcomponentsallocatedtobuild3includedtheexternal
communicationsinterfaceprotocolhandling,thecompleteduser-systeminterface
forthemissionoperators,theuser-systeminterfaceforthecomputersupport
positions,thesystemservicesformissionreconfigurations,databaseresets,off-line
datareduction,andthenucleardetonationalgorithms.
•Build4-Thisbuildprovidedthefinalinstallmentofmissilewarningalgorithmsfor
satelliteearlywarningsystems,thefinalinstallmentofmissionmanagementand
missionstatusdisplays,andthefinalinstallmentofexternalcommunications
interfaceprocessing.
•Build5-InthemiddleoftheCommonSubsystemschedule,build5wasaddedto
coincidewithaparticularexternalinterface(beingbuiltonaseparatecontract),the
scheduleforwhichhadslippedsothatitwasnotgoingtobeavailableduringits
originallyplannedbuild(build4).Consequently,theexternalinterfacewas
scheduledintoanentirelynewbuild.

18. 5.2 Incremental Design Process
•The individual milestones within a build included a
•preliminary design walkthrough(PDW),
•critical design walkthrough (CDW),
•turnover review (TOR).
•The schedules for these milestones were flowed down from, and integrated with, the
higher level project milestones (SRR, IPDR, PDR, and CDR). Figure provides an
overview of a build's life cycle and the focus of activities.

19. 5.2 Incremental Design Process
•Well-definedsequenceofdesignwalkthroughswereinformal,detailed,technicalpeer
reviewsofintermediatedesignproducts.
•Theywereattendedbyinterestedreviewers,includingotherdesigners,testers,andeven
stakeholdersoutsidethesoftwareorganization(customer,user,projectsystems
engineeringpersonnel).
•Attendancewasusuallykepttoasmallnumberofknowledgeablepeople,typically10to
20.
•Theexplicitfocusofthesereviewswastheimportantcomponents,thearchitectural
interfaces,andthedrivingissuesnamely,the20%ofthestuffthatdeservedscrutiny.
•PDWsandCDWsusuallylastedoneortwodays,witheachoftheparticipatingCSCI
managersresponsibleforpresentingappropriatematerial.
PreliminaryDesignWalkthroughs
•Thewalkthroughfocusedonthestructuralattributesofthecomponentswithinthebuild.
•Overview:CSCIoverview,interfaces,components,andmetrics
•Components:walkthroughofeachmajorcomponent,showingitssourcecodeinterface,
allocatedsystemrequirementsspecification(SRS)requirements,currentmetrics,
operationalconceptforkeyusagescenarios,standalonetestplan,anderroneous
conditionsandresponses
•Demonstration:focusedonexercisingthecontrolinterfacesacrossthecomponents
withintheintegratedarchitecture.

20. 5.2 Incremental Design Process
CriticalDesignWalkthroughs
•WhilethePDWfocusedonthedeclarativeviewofthedesign,theCDWfocusedonthe
completenessofthecomponentsandthebehavioralperspectiveofoperatingwithinthe
allocatedperformancerequirements.
•CSCIoverview:interfaces,components,andmetrics;summaryofchangessincePDW;
dispositionofallPDWactionitems;buildintegrationtestscenarios.
•Components:walkthroughofeachmajorcomponent,showingitssourcecodeinterface,
allocatedSRSrequirements,currentmetrics,operationalconceptforkeyusagescenarios,
stand-alonetestplan,anderroneousconditionsandresponses.
•Demonstration:focusedonexercisingthecriticalperformancethreads.
CodeWalkthroughs
•Better dissemination of self-documenting source code style
•Identification of coding issues not easily caught by compilers and source code analysis
tools
•Object naming, coding style, and commenting style:Does it promote readability?
•Unnecessarily complex objects or methods:Are there simpler approaches?
•Reuse: Is custom software being built where reusable components exist?
•Potential performance issues:Are there potentially inefficient implementations?
•Reduction of the amount of source code needed for review in the larger design
walkthroughs
•Exposure of inexperienced personnel to the products of experts and vice versa

21. 5.2 Incremental Design Process
TurnoverReviews
•Turnoverreviewswerenotreallyreviews.Theyweretypicallya
one-monthactivityduringwhichcomponentswerecompletedwith
stand-alonetestingandturnedoverforconfigurationcontrol,build
integrationtesting,andengineeringstringtesting.
•ThecheckpointsusedonCCPDS-Rfortheincrementaldesign
processaregoodexamplesoftheminormilestones.
•Themixtureofwalkthroughsandinspectionswasfocusedonthe
20%ofcomponentsthathadapotentialhighreturnonhuman
resourceinvestment.
•Ingeneral,therealvalueofdesignwalkthroughsandinspections
wascommunicationsamongprojectsubteamsandmethodical
coordinationofprocesses.
•Veryfewseriousqualityflawswereuncoveredinthesemeetings
(asopposedtothedemonstrationactivities),butthetechnical
interchangewaswellmechanizedbythesecheckpoints.

22. 5.3 COMPONENT EVOLUTION
•TBD(ToBeDefined)packageincludespredefinedTBDtypes,TBDconstants,TBD
values,andaTBDprocedurefordepictingsourcelinesofcodeassociatedwithcomments
thattogetherwouldactasplaceholdersforas-yetundefinedcodesegments.
•Proceduredeclaration:
TBD_Statements(Number_of_Statements:InInteger)
Thebasiccomponentevolutionwouldbe:
•Atcreation,onlytheinterfacewouldbedefinedwithAdasourcelinesandcorresponding
comments.TheestimatedSLOCcountforthecomponentwouldtypicallybespecifiedby
asingleTBD_Statementsline.
•AtPDW,thesubstructureofthecomponentwouldbefleshedoutalongwithmost
componentdeclarationsandestimatesofthesubordinateprogramunitsusingmultiple
callstoTBD_Statements.Atthistime,therewouldgenerallybeabout30%oftheSLOC
inAdaand70%inADL.
•ByCDW,mostoftheprogramunitinterfacesanddeclarationswouldbefullyfleshedout
inAda,withsomedetailedprocessingstillusingTBD_Statementsasplaceholders.CDW-
Ievelcomponentswouldbeabout70%Adaand30%ADL.Aguidelinealsostatedthatby
CDW,therewouldbenocallstoTBD_Statementswithvaluesgreaterthan25.
•Byturnover,thestringTBDwouldnotappearanywhereinthesourcefiles.Thiswould
correspondtoacompleteimplementation.

23. 5.4 Incremental Test Process
Testsequencewasdevisedwithfivedifferenttestactivities.stand-alonetest,build
integrationtest,reliabilitytest,engineeringstringtest,andfinalqualificationtest.
1.Stand-AloneTest(SAT)-Thedevelopmentteamswereresponsibleforstandalonetesting
ofcomponentsbeforedeliveryintoaformal,configurationcontrolledtestbaselineusedfor
allothertestactivities.SATtypicallytestedasinglecomponentinastand-alone
environment.
Thisleveloftestingcorrespondstocompletenessandboundaryconditionteststotheextent
possibleinastandalonecontext.
2.BuildIntegrationTest(BIT)-Thiswasmostlyasmoketesttoensurethatpreviously
demonstratedcapabilitiesstilloperatedasexpected.ABITsequenceistheprimaryquality
assessmentvehicleforclosingoutaturnoverreview.
Agivenbuildturnovermaytakedaysorweeks,dependingonitssizeorthepercentageof
newcomponentry.ThepurposeofBITisnottoverifyrequirementsbuttoestablishastable,
reliablebaseline.
Itisveryinformal,dynamic,andfocusedonexposingerrorsandinconsistencies.
BITsvalidatethefollowing:
•Previouslydemonstratedthreadscanberepeatedsuccessfully.
•Previouslydefineddeficiencieshavebeenresolved.
•Interfacesacrosscomponentsarecompletelytested.
•Thebaselineisstableenoughforefficientrequirementsverificationtesting.

24. Incremental baseline evolution and test activity flow

25. 5.4 Incremental Test Process
3.ReliabilityTest-OneoftheoutputsoftheBITprocessandaturnoverreview
wasastabletestbaselinethatwassubjectedtoextensiveafter-hoursstresstesting
forlongperiodsoftimeunderrandomizedbutrealistictestscenarios.Thiswas
designedtohelpuncoverpotentiallyinsipid,transienterrorsofmajordesign
consequence.Reliabilitytestingloggedasmuchtesttimeaspossiblewhile
resourceswereotherwisemostlyidle(onnightsandweekends}.
4.EngineeringStringTest(EST)-Thesetestsfocusedonverifyingspecific
subsetsofrequirementsacrossmultipleCSClsthroughdemonstrationandtestof
usecaserealizations(calledcapabilitythreads).
5.FinalQualificationTest(FQT)-ThesetestswereequivalenttoESTsexceptthat
theyrepresentedthesetofrequirementsthatcouldnotbeverifiedunlessthe
wholesystemwaspresent.Forexample,a50%reservecapacityrequirement
couldnotbeverifieduntilFQT,whenthewholesystemwasoperational

26. 5.5 DOD-STD-2167 A ARTIFACTS
•UseoftheevolvingAdasourcefilesasthesinglehomogeneouslife-cycledesignformat
andevolutionofthesefilesinaself-documentingmanner.ThistechniqueexploitedAda's
readabilityfeaturesandavoidedtheextraeffortinvolvedinpreparingseparate,detailed
designdescriptionsthatinevitablydivergefromtheimplementation.
•Organizationofthetestsequencesanddeliverabledocumentsaroundthebuildcontent
drivenbysubsetsofusecasesratherthanbyCSCI.Thisstring-basedtestingspanned
componentsinmultipleCSCIs.Itwasorganizedbybuildandmechanizedviaasoftware
testplan,softwaretestprocedure,andsoftwaretestreportdocumentationsequence.These
documentsequenceswereprovidedforeachBIT,eachEST,andFQT(onefinalall-
encompassingtestsequence).Eachtestsequenceinvolvedcomponentsfromseveral
(incomplete)CSCIsbecauseintegrationwasproceedingcontinuously.
•Buildingofseparateunittestdocumentationasself-documented,repeatablesoftware.This
wastreatedlikeotheroperationalsourcecodesothatitwasmaintainedhomogeneouslyand
up-to-dateforautomatedregressiontesting.ThesameconceptwasusedfortheBITand
ESTscenariotesting:
Ratherthandeveloptestproceduredocuments,theCCPDS-Rprocessgeneratedself-
documentingtestscenariosthatweresoftwareprogramsintheirownright.Becausethey
weresubjectedtochangemanagementjustlikeothersoftware,theywerealwaysmaintained
up-to-dateforautomatedregressiontesting.

27. CCPDS-R software artifacts

28. Software development file evolution

29. 5.7 DEMONSTRATION-BASED ASSESSMENT
•Designreviewonanexecutabledemonstrationprovidesa
moreunderstandableandconcretereviewvehicleforadiverse
setofstakeholders.
•Demonstrationsprovideacuteinsightintotheintegrityofthe
architectureanditssubordinatecomponents,therun-time
performancerisks,andtheunderstandingofthesystem's
operationalconceptandkeyusecases.
•Thesequenceofdemonstrationactivitiesincludedthe
developmentofaplan,definitionofasetofevaluationcriteria,
integrationofcomponentsintoanexecutablecapability,and
generationoftestdrivers,scenarios,andthrow-away
components.
•Thedemonstrationplansarecapturedthepurposeofactual
evaluationcriteriaforassessingtheresults,thescenariosof
execution,andtheoverallhardwareandsoftware
configurationthatwouldbedemonstrated.

30. 5.7 DEMONSTRATION-BASED ASSESSMENT
KeylessonswerelearnedintheCCPDS-Rdemonstrationactivities:
•EarlyconstructionoftestscenarioshasahighROI.Theearlyinvestmentin
buildingsomeofthecriticaltestscenariosservedtwoinvaluablepurposes.
•Demonstrationplanningandexecutionexposetheimportantrisks.
Negotiatingthecontentofeachdemonstrationandtheassociated
evaluationcriteriaservedtofocusallteamsinorganization.
•DemonstratedefinedcapabilitiesatNORAD
•Demonstrationinfrastructure,instrumentation,andscaffoldinghaveahigh
ROI.
•CapabilitieswellunderstoodbythecustomerandTRW
•Demonstrationactivitiesexposethecrucialdesigntrade-offs,The
integrationofthedemonstrationprovidedtimelyfeedbackontheimportant
designattributesandthelevelofdesignmaturity.
•Earlyperformanceissuesdriveearlyarchitectureimprovements.
•Resultswereapttochangerequirements,plansanddesigns

31. IPDR Demonstration
InterimPDRmajormilestonedemonstrationoftheCommonSubsystemhadthreecritical
objectives:
1.Tangibleassessmentofthesoftwarearchitecturedesignintegritythrough
constructionofaprototypeSAS
2.Tangibleassessmentofthecriticalrequirementsunderstandingthroughconstruction
oftheworst-casemissilewarningscenario.
3.Exposureofthearchitecturalrisksassociatedwiththepeakmissilewarningscenario
andthefaultdetectionandrecoveryscenario.
•FiguresummarizestheschedulefortheIPDRdemonstrationwithintenseintegration
periodinthetwomonthsbeforethedemonstration.
•Thefirstthreemonthsofplanning,whichencompassedadraftplan,governmentreview
andcomment,andfinalplanproduction,couldhavebeenachievedinoneweekwitha
collocatedteamofallinterestedstakeholders.
•Thereviewsequencethatoccurredwasarequirementofthecontract.Becausethiswasthe
firsttimethatTRWorthecustomerhadusedademonstration-basedapproach,bothwere
unsureofthebestprocessandagreedonanoverlyconservativeapproach.
•Thisdemonstrationwasthefirstattemptatconstructingafull-scaleSAS.
•Thesubsequentdemonstrationstendedtohaveshorter,butequallyintense,integration
activitieslasting4or5weeks.

32. CCPDS-R first demonstration activities and schedule

33. IPDR Demonstration Scope
•Thecontractorshalldemonstratethe(0110wingcapabilitiesattheNORADDemo1:
systemservices,systeminitialization,system)failoverandrecovery,system
reconfiguration,testmessageinjection,anddatalogging.
•ThesecapabilitieswerefairlywellunderstoodbythecustomerandTRW.
1.SystemservicesweretheNASsoftwarecomponentsofgeneralutilitytobereusedacross
allthreesubsystems.Thesecomponentsareinter-processcommunicationsservices,generic
applicationscontrol(generictaskandprocessexecutives),NASutilities(listroutines,
nameservices,stringservices),andcommonerrorreportingandmonitoringservices.
Theseareneededtodemonstrateanyexecutablethread.
2.Datalogging(SSVCSCI)wasacapabilityneededtoinstrumentsomeoftheresultsofthe
demonstrationandwasaperformanceconcern.
3.Testmessageinjection(TASCSCI)componentspermittedmessagestobeinjectedintoany
objectinthesystemsothattherewasageneraltestdrivercapability.
4.Systeminitializationillustratestheexistenceofaconsistentsoftwarearchitectureskeleton
anderror-freeoperationofasubstantialsetofthesystemservices.Initializesalarge
distributedsoftwarearchitecturewithcommercialcomponentsinagiventime.
5.Thesecondscenario(phase2)wastoinjectthepeakmessagetrafficloadintothe
architectureandcausealltheinternalmessagetraffictocascadethroughthesystemina
realisticway.Executingthisscenariorequiredallthesoftwareobjectstohavesmart,but
simple,messageprocessingstubstobe"modeled“.

34. IPDR Demonstration Scope
•Systemfailoverandrecovery(phase3)wasoneoftheriskiestscenarios,becauseit
requiredaverysophisticatedsetofstatemanagementandstatetransitioncontrolinterfaces
tobeexecutedacrossalogicalnetworkofhundredsofsoftwareobjects.
•Thebasicoperationofthisusecasewastoinjectasimulatedfaultintoaprimarythread
operationalobjecttoexercisethefollowingsequenceofevents:faultdetection,fault
notification,orchestratedstatetransitionfromprimarythreadtobackupthread,shutdown
ofprimarythread.Allthesenetworkstatetransitionsneededtooccurwithoutinterruption
ofservicetothemissilewarningoperators.
•Reconfiguration,inthisspecificcase,meantrecoveringfromadegradedmode.
•Followingthesystemfailoverdefinedabove,anewbackupthreadwouldbeinitializedso
thattherewasminimumexposuretosingle-pointfailures.Inthedeliveredsystem,repair
immediatelyfollowedfailover.

35. IPDRDemonstrationEvaluationCriteria
•Derivedfromtherequirements,theassessments,andtheevolvingdesigntrade-offs:
Allphases:
•Nocriticalerrorsshalloccur.
Phase1:
•Thesystemshallinitializeitselfinlessthan10minutes.
•Thesystemshallbeinitializedfromasingleterminal.
•Afterinitializationiscomplete,thenumberofprocesses,tasks,andsocketsshall
matchexactlytheexpectednumbersinthethen-currentSASbaseline.
Phase2:
•Averagedovertheworst-caseminuteofthe20-minutepeakscenario,thetotal
processorutilizationforeachnodeshallbelessthan30%.
•Thereshallbenoerrorreportsofduplicateorlostmessages.
•Alldisplayeddatashallbereceivedwithin1secondfromitsinjectiontime.
•Themessageinjectionprocessshallmaintainaninjectionratematchingtheintended
scenariorate.
•Thedatalogsshallshownounexpectedstatetransitionsorerrorreportsandshalllog
allinjectedmessages.

36. IPDRDemonstrationEvaluationCriteria
Phase3:
•Theoperatorshallbecapableofinjectingafaultintoanyobject.
•Anerrorreportshallbereceivedwithin2secondsoftheinjectionofafault.
•Theswitchoverfromtheprimarytobackupthreadshallbecompletedwithin2
secondsofthefaultinjectionwithnolossofdata.
•Theshutdownofthefailedprimarythreadandreinitializationasanewbackupthread
shallbecompletedinlessthan5minutesfromfailure.
•Thedatalogsshallmatchtheexpectedstatetransitionswithnofatalerrorsreported
otherthantheinjectedfault.

37. IPDRDemonstrationResults
•Fromthe37evaluationcriteria,31wereconsideredsatisfactory.
•Sixcriteriawerenotmet,includingthreeoftheessentialcriteriajustdiscussed.Thesewere
consideredveryseriousissuesthatrequiredimmediateredesignandre-demonstration.
•Ofmostconcernwasexcessiveprocessorutilizationduringthepeakloadscenario.While
thethresholdwas30%,actualutilizationwas54%.
•Thiscorrespondedtotheessentialoverheadofthearchitecturalinfrastructure,operating
system,andnetworkingsoftware.BecausethiswasalwaysaperceivedriskoftheCCPDS-
Rreusablemiddlewaredesign,itreceivedextensiveattention.
•Fivedistinctactionitemsforperformanceanalysiswerecreated,aswellasanactionitem
todemonstratetheperformanceimprovementatthenextprojectmanagementreviewafter
thefiveactionitemswereresolved.
Fiveactionitemsare
1.Updatethescenario.
2.Tunetheinterprocesscommunications.
3.Enhancenetworktransactions.
4.ImproveperformanceoftheIPecomponent.
5.ImprovereliabilityintheIPecomponent.

38. IPDRDemonstrationResults
Lessonswerelearnedfromresultsare:
1.Veryeffectivedesignreviewwasoccurringthroughouttheperiod.Thedemonstration
wastheresultoftheengineeringteam'sreview,presentedtothestakeholdersas
tangibleevidenceofprogress.Althoughweendedupwithonlyfiveopenissues,50or
moredesignissueshadbeenopened,resolved,andclosedduringthe8-week
integrationactivity.Thisearlyresolutionofdefects-intherequirementsspecification,
theprocess,thetools,
2.Throughday-to-dayparticipationinthisactivity,gaineddetailedinsightintowhere
thedesignwasweak,whereitwasrobust,andwhy.
3.Thedemonstrationservedasastrongteam-buildingexerciseinwhichtherewasa
verytangiblegoalandtheengineerswereworkingintheforumtheypreferred:getting
stufftowork.
4.Thedetailedtechnicalunderstandingandobjectivediscussionsofdesigntrade-offs
provedinvaluabletodevelopingatrustworthyrelationshipwithallstakeholders,
includingthecustomer,theuser;andTRWmanagement.Wewerearmedwithfacts
andfigures,notsubjectivespeculation.

39. Government Response to the IPDR Demonstration
AftercarefulevaluationoftheGovernment'sPreliminaryDemo1Plancomments,the
followingobservationssummarizethissubmittaloftheDemo1Planandthemodificationsthat
havebeenmadefromthepreviousversion:
1.Thissubmittalhaseliminatedallrequirementsreferencestoavoidmakinganyallusionto
anintentofsatisfying,proving,or.demonstratinganyrequirements.Theserequirements
verificationactivitiesareperformedbythetestorganizationinaveryrigorousand
traceablefashion.Thedemonstrationactivityisintendedtobeanengineering-intensive
activity,stream-linedthroughminimaldocumentation,toprovideearlyinsightintothe
designfeasibilityandprogress.TRWintendstomaximizetheusefulnessofthe
demonstrationasanengineeringactivityandtoavoidturningitintoalessuseful
documentation-intensiveeffort.
2.Severalgovernmentcommentsrequestedfurtherdetailsonrequirements,designs,etc.This
informationisnotnecessaryintheDemoPlan.Itisredundantwithotherdocuments(SRS,
SDD,designwalkthroughpackages)oritisprovidedintheinformaltestprocedures
delivered2weekspriortothedemonstration.Providingmoreinformationinasingle
document(andineverydocument)maymakethereviewer'sjobeasierbutitwouldalsobe
excessive,moretime-consuming,andcounterproductivetoproduce,therebyreducingthe
technicalcontentoftheengineeringproductbeingreviewed.

40. Government Response to the IPDR Demonstration
3.Inlightofthegovernment'sconcernovertherelationshipofthedemonstrationtothe
requirements,theevaluationcriteriaprovidedinthisplanshouldbecarefullyscrutinized.
Wefeelthattheevaluationcriteriaareexplicit,observable,andinsightfulwithrespectto
determiningdesignfeasibility,especiallyatsuchanearlypointinthelifecycle.Although
weareopentoconstructivemodificationoftheseevaluationcriteria,wefeelthat
modifyingthemtorelatemorecloselytotheSystemSpecificationorSRSrequirements
wouldbeinappropriate.Therequirementsperspectiveandourdemonstrationperspective
aredifferentanddifficulttorelate.
4.Thesourcecodeforthecomponentsbeingdemonstratedhasnotbeendeliveredwiththe
planasrequiredinthestatementofwork.Thetotalvolumeforthedemonstrated
componentsisroughly1to2feetthick,anditisstillchangingatarapidrate.Insteadof
deliveringallthesourcecode,interestedreviewersmayrequestspecificcomponentsfor
review.Allsourcecodewillbebrowseableatthecontractorfacilityduringthe
demonstration.

41. CORE METRICS
TRWformulatedametricsprogramwithfourobjectives:
•Providedataforassessingcurrentprojecttrendsandidentifyingtheneedfor
managementattention
•Providedataforplanningfuturebuildsandsubsystems
•Providedataforassessingtherelativecomplexityofmeetingthesoftwareend-
itemqualityrequirements
•Providedataforidentifyingwhereprocessimprovementsareneededand
substantiatingtheneed
1.DEVELOPMENT PROGRESS
•TheAda/ADLmetrics.Thesedataprovidedgoodinsightintothedirectindicators
oftechnicalprogress.Bythemselves,thesemetricswerefairlyaccurateat
depictingthetrueprogressindesignandimplementation.Theyweregenerally
weakatdepictingthecompletedcontractdeliverablesandfinancialstatus.
•Earnedvaluemetrics.Thesedataprovidedgoodinsightintothefinancialstatus
andcontractdeliverables.Theyweregenerallyweakindicatorsoftruetechnical
progress.

42. Development progress summary
•Figuredisplaysmonth17status:Build2SATtestingisonemonthbehindschedule,build3
designworkisonemonthaheadofschedule,theCommonSubsystemdesigneffortison
schedule,andCommonSubsystemSATtestingisonemonthbehindschedule.
•Theplannedevolutionwasbasedroughlyonweight-averagingtheSLOCcountsforeach
buildwiththeguidelinesdescribedinSection0.5.3:30%donebyPDWand70%doneby
CDW.Overall,theCommonSubsystemperformedveryclosetoitsplan,withoneexception.

43. Common Subsystem development progress

44. CORE METRICS
2.TESTPROGRESS
•Buildintegrationtestsandrequirementsverificationtesting(someSATs,ESTs,andFQT).
Buildintegrationtestingprovedtobelesseffectivethanexpectedforuncoveringproblems.
•TableandFigureprovideperspectivesontheprogressmetricsusedtoplanandtrackthe
CCPOS-Rtestprogram.Thefigureplotstheprogressagainsttheplanforrequirements
verificationtests.
•SATs,ESTs,andFQTsweresourcesoftestcasesusedbythesoftwareorganization.

45. •SATsweretheresponsibilityofthedevelopmentteamsbuthadtobeexecutedintheformal
configurationmanagementenvironmentandwitnessed(peer-reviewed)bythetest
personnel.
•ESTsconsistedoffunctionallyrelatedgroupsofscenariosthatdemonstratedrequirements
spanningmultiplecomponents.FQTsweretestsforrequirementscompliancethatcouldnot
bedemonstrateduntilacompletesystemexisted.
•Quantitativeperformancerequirements(QPRs)spannedallCSCIs.TheSATprocedures
wererarelyavailable30to60daysbeforeturnover,asrequiredbythecontractforany
requirementsverificationtest.TheformalSATprocesswasoneofthemainreasonsthatthe
turnoverreviewswereconsistentlycompletedlaterthanplanned.

46. CORE METRICS
3.STABILITY
•Figureillustratestheoverallrateofconfigurationbaselinechanges.Itshowsthecumulative
numberofSLOCthatwerebrokenandthenumberofSLOCrepaired(checkedbackinto
thebaselinewithfixes,enhancements,orotherchanges).
•Breakageratesthatdivergedfromrepairratesresultedinmanagementattention,
reprioritizationofresources,andcorrectiveactionstakentoensurethatthetest
organization(drivingthebreakage)anddevelopmentorganization(drivingtherepair)
remainedinrelativeequilibrium.Overall,thesituationshowndepictsaveryhealthyproject.

47. CORE METRICS
4.MODULARITY
•Figureidentifiesthetotalbreakageasaratiooftheentiresoftwaresubsystem.Thismetric
identifiesthetotalscrapgeneratedbytheCommonSubsystemsoftwaredevelopment
processasabout25%ofthewholeproduct.Industryaveragesforsoftwarescrapruninthe
40%to60%range.Theinitialconfigurationmanagementbaselinewasestablishedaround
thetimeofPDR,atmonth14.Therewere1,600discretechangesprocessedagainst
configurationbaselines.

48. CORE METRICS
5.ADAPTABILITY
•OvertheentireCommonSubsystem,about5%ofthetotaleffortwasexpendedin
reworkactivitiesagainstsoftwarebaselines.Theaveragecostofchangewasabout24
hoursperSCO.Thesevaluesprovidesomeinsightintotheeasewithwhichthesoftware
baselinescouldbechanged.
•ThelevelofadaptabilityachievedbyCCPDS-Rwasroughlyfourtimesbetterthanthe
typicalproject,inwhichreworkcostsoverthedevelopmentlifecycleusuallyexceed20%
ofthetotalcost.FigureplotstheaveragecostofchangeacrosstheCommonSubsystem
schedule.The1,600+SCOsprocessedagainsttheevolvingconfigurationbaselinebyFQT
resultedinafairlystablecostofchange.

49. CORE METRICS
6.MATURITY
•CCPDS-Rhadaspecificreliabilityrequirement,forwhichthesoftwarehadaspecificallocation.
Theindependenttestteamconstructedanafter-hours,automatedtestsuitethatexercisedthe
evolvingsoftwarebaselineswithrandomizedmessagescenarios.
•Thisstrategyresultedinextensivetesttimebeingloggedunderrealisticconditionsfromwhicha
crediblesoftwareMTBFcouldbesubstantiated.Thereliability-criticalcomponents,forcedby
theiterationplanintotheearliestbaselines,weresubjectedthemostreliabilitystresstesting.
Figureillustratestheresultswithmoderndistributedarchitecturesuncoveringsignificant
issuesofraces,deadlocks,resourceoverruns,memoryleakage,andotherHeisen-bugs.
Executingrandomizedandacceleratedscenariosforlongperiodsoftime(runningallnightor
overalongweekend)enablesearlyinsightintooverallsystemresourceintegrity.

50. CORE METRICS
7.COSTEFFORTEXPENDITURES BYACTIVITY
•TableprovidestheoverallcostbreakdownfortheCCPDS-RCommonSubsystem.These
datawereextractedfromthefinalWBScostcollectionrunsandwerestructured.

51. •Theoveralltestteameffortisrelativelylowcomparedwiththeefforttypically
expendedonprojectsthatusedtheconventionalprocess.
•CCPDS-Rusedanefficientenvironmentthatrepresented11%ofthetotalcostof
theeffort.
•Overallmaintenance(totalreworkeffortexpended)wasonly5%ofthetotalcost.
Althoughthisisnotshownexplicitlyinthetables,itwastrackedintheindividual
CSCIWBSelements.

52. FewothermetricsofCCPDS-Rprojectperformance:
1.softwaresizeevolution,
2.subsystemprocessimprovements,
3.SCOresolutionprofile,and
4.CSCIproductivitiesandqualityfactors.
1.Softwaresizeevolution
•Largeamountofcodegrowthfromtheoriginalcontractbid(150,000SLOC)tothedelivered
product(355,000SLOC),withnosubstantialincreaseinthesoftwaredevelopmentbudget.
Thereweretworeasonsforthislevelofcodegrowth:
•Themethodforcountingsourcelineswaschangedaroundmonth8toprovideabetter
balanceinestimatingtheengineeringeffortandtobeconsistentwiththecountingmethod
embracedbyAdaCOCOMO.
•Severalautomaticcodegenerationtoolsweredevelopedthatoutputverbosesourcecode
withfewerhuman-generatedinputlines.Thesetoolswereusedforthestraightforward
generationofdisplayformats,messagevalidationprocessing,andsocket/circuitbookkeeping
functions.Theyrepresentedabout14,000SLOCoftools,whichrequiredanother20,000lines
ofinputdatafiles.Theoutputofthesetoolswasabout200,000SLOCofoperational
software.Ingrossterms,thesecodegenerationtoolsresultedinaboutafivefoldreturnon
investment.
8. OTHER METRICS

53. FourcodingstandardsallowedtheSLOCcountingtobeconsistent:
1.Eachparameterofasubprogramdeclarationislistedonaseparateline.Theeffort
associatedwithdesignofasubprograminterfaceisgenerallyproportionaltothenumberof
parameters.
2.Forcustomenumerationtypes(suchassocketnamesandsystemstates)andrecordtypes,
eachenumerationorfieldislistedonaseparateline.Customtypesusuallyinvolvecustom
designandengineering,resultinginanincreasednumberofSLOC.
3.Forpredefinedenumerationtypes(suchaskeyboardkeysandcompassdirections),
enumerationsarelistedonthefewestnumberoflinespossiblewithoutlossofreadability.
Thesetypesgenerallyrequirenocustomengineering.
4.Initializationofcompositeobjects(suchasrecordsandarrays)islistedwithonecomponent
perline.Eachoftheseassignmentsrepresentsacustomstatement;an"others"clauseis
typicallyusedfornoncustomassignments.
2.SUBSYSTEMPROCESSIMPROVEMENTS
•Overall,theCommonSubsystemsubsidizedmuchofthegroundworkforthePDSand
STRATCOM subsystems-namely,theprocessdefinition,thetools,andthereusable
architectureprimitives.
•Witheachsuccessivesubsystem,productivityandqualityimprovedsignificantly.
•Thisistheexpectationforamaturesoftwareprocesssuchastheonedevelopedandevolved
onCCPDS-R.ThePDSSubsystemwasdeliveredat40%ofthecostperSLOCofthe
CommonSubsystem,andtheSTRATCOMSubsystemat33%.

54. 3.seaRESOLUTIONPROFILE
•Theaveragechangecostsevolvedovertimeintoafairlyconstantvalueof16hoursper
change.Thiseffortincludedanalysistime,redesign,recode,andretestoftheresolution.
4.CSCIPRODUCTIVITIESANDQUALITYFACTORS
•ProductivitiesfortheCSCIsarenotabsolute.Thesubsystemproductivityisbasedonatotal
effortofapproximately1,800staff-months.Thisincludesallmanagement,development,
andtestresources.TheindividualproductivitiesofeachCSCIwerenormalized.
Productivitiesaredescribedfromtwoperspectives:SLOCperstaff-monthandESLOC
perstaff-month.
Thesedataleadstothefollowingconclusions:
•NASwasanextremelycomplexsoftwareengineeringproblem,requiringandachieving
bothhighperformanceandreusability.Ithadanexceptionalteam,wasbasedonanexisting
prototype,andhadadequateschedule.
•SSVhadveryhighabsoluteproductivitybecausetheautomaticallygeneratedcode,from
customCASEtools,wascontainedmostlywithinthisCSCL.Theabove-averageteamon
SSValsocontributedtothehighproductivity.
•DCOwasfairlyaverageonallcountsbutaccommodatedsubstantialrequirementsvolatility
inthedisplayinterfacewithoutacontractamendment.ThedesignofthisCSCIandthe
performanceoftheteamwerefarbetterthanthesenumberswouldindicate.

55. •TAShadaverylowproductivitydespitebeingthesimplestandmostwellunderstood
software.ThemainreasonwasfarlessambitiousthantheplansforotherteamsandTAS
teamwaslocatedoff-site,withhighlyconstraineddevelopmentenvironmentresources.
•CMPhadaveryhighcostofchangeandlowproductivityfornoobvioustechnicalreason.
Toensuretechnicalintegrity,theinherentmissilewarningalgorithmchangeswereclosely
scrutinizedbymanystakeholders.Thecoordinationofthisprocessresultedinveryhigh
overheadinCMPproductivityandchanges.
•CCOhadtheworstqualitymetrics,Thiswasdueprimarilytoadesignthatdidnotforeseea
majormessagesetchangeandthereforeresultedinfairlybroadandhard-to-resolve
breakage.TheCCOteamwasalsoperhapsthemostdifficulttotransition(culturally)tothe
process,metrics,anddemonstrationapproachusedonCCPDS-R.

56. 9. People factors
CCPDS-Rusedtwouniqueapproachestomanagingitspeople.
•Thefirstwasthecoreteamconcept,whichfocusedonleveragingtheskillsofafewexperts
acrosstheentireteam.
•Thesecondwastargetedatactivelyavoidingattrition.
•CCPDS-RwasTRW'sfirstlargeAdaproject,andmanagementwasconcernedthatpersonnel
trainedbytheprojectwouldbecomeattractivetargetsforopportunitieselsewhereinsideand
outsidethecompany.
•AsaresultoftheoverallmanagementapproachtoCCPDS-R,therewasverylittleattritionof
peopleacrosstheCommonSubsystem,withmostoftheengineeringteamtransitioningtonew
assignmentsatplannedpointsinthelifecycle.
•Contrarytoinitialexpectations,thePDSandSTRATCOMsubsystemswereoverlappedenough
withtheCommonSubsystemthatmostofthepeopleemployedwerenewtotheproject.
•Coreteamconcept
•Leverageskillsofafewexpertsacrosstheentireteam
•Avoidattrition
•Profitsharingofawardfees

57. 9.1 Core Team
1.Developingthehighestleveragecomponents(mostlywithintheNASCSCI).Thesecomponents
resolvedmanyofthedifficultcomputerscienceissuessuchasreal-timescheduling,interprocess
communications,run-timeconfigurationmanagement,errorprocessing,anddistributedsystems
programming.Asaresultofencapsulatingthesecomplexissuesinasmallnumberofhigh-leverage
components,themainstreamcomponentsweresimplerandfarlessdependentonexpertpersonnel.
2.Settingthestandardsandproceduresfordesignwalkthroughsandsoftwareartifacts.In
general,thecoreteamrepresentedthefrontlinepioneersformostofthesoftwareactivities.This
teamwasgenerallythefirstteamtoconductanygivenprojectworkflowandbuiltthefirstversion
ofmostartifacts.Consequently,thecoreteamwasintimatelyinvolvedwithsettingprecedent,
whetheritwasthestandardsforagivenactivityortheformat/contentofagivenartifact.
3.Disseminatingtheculturethroughoutthesoftwareorganization.Thecoreteamwastrulya
single,tight-knitteamduringtheinceptionphaseandformostoftheelaborationphase.Asthe
processandarchitecturestabilized,theteamstartedtomigrate,withseveralofitsmemberstaking
ontechnicalleadershiprolesonthevariousdevelopmentandassessmentteams.During
constructionandtransition,afewmembersofthecoreteamstillmaintainedthearchitecture
integrityacrosstheentireproject.However,therewasalsoasetofgloballymindedindividuals
withstrongrelationshipstothearchitectureteamwhobecameimmersedinotherareasof
developmentandassessment.Theseteamandpersonneltransitionsprovedtobeaninvaluable
mechanismformaintainingacommonculture.

58. 9.2 AWARD FEE FLOWDOWN PLAN
•Awardfeeflowdownplanwasthatemployeeswouldshareintheprofitabilityoftheproject.
TRWmanagementagreedtoallocateasubstantialportionoftheawardfeepoolateachmajor
milestonetobegivendirectlytoprojectemployees.Thisadditionalcompensationwastobe
distributedtotheindividualsbasedontheirrelativecontributionandtheirlongevityonthe
project.
Theimplementationoftheawardfeeflowdownplanwasintendedtoachievethefollowing
objectives:
•Rewardtheentireteamforexcellentprojectperformance
•Rewarddifferentpeergroupsrelativetotheiroverallcontribution
•Substantiallyrewardthetopperformersineverypeergroup
•Minimizeattritionofgoodpeople
Thiswasthebasicoperationalconceptoftheplan:
•Managementdefinedthevariouspeergroups(systemsengineering,softwareengineering,
businessadministration,andadministration).
•Every6months,thepeoplewithineachpeergrouprankedoneanotherwithrespecttotheir
contributiontotheproject.Themanagerofeachpeergroupalsorankedtheentireteam.The
managercompiledtheresultsintoaglobalperformancerankingofthepeergroup.
•Eachawardfeewasdeterminedbythecustomeratcertainmajormilestones.Halfofeach
awardfeepoolwasdistributedtoprojectemployees.
•Thealgorithmfordistributionstoprojectemployeeswasfairlysimple.Thegeneralrangeof
additionalcompensationrelativetoeachemployee'ssalarywasabout2%to10%eachyear.

59. 10. CONCLUSIONS
•TRW and the Air Force have extensively documented the successes of architecture-first
development on CCPDS-R.
•This project achieved twofold increases in productivity and qualityalong with on-budget,
on-schedule deliveries of large mission-critical systems.
•The success of CCPDS-R is due, in large part, to the balanced use of modern technologies,
modern tools, and an iterative development process that is substantially
•OneoftheprimaryimprovementsthatwasenabledbytheCCPDSRsoftwareapproachwas
theteamworkbetweenthecustomer,user,andcontractor.Continuousbartering,negotiation,
andinterpretationofthecontractdeliverableswereproductiveinmakingrealprogressand
ensuringthateachphaseofthelifecycleresultedinawin-winsituationforallstakeholders.
•CCPDS-Rmaintainedgoodprojectperformanceandnonadversarialrelationshipsamong
stakeholdersthroughoutthelifecyclewhilecontinuouslyabsorbingamoderatelevelof
requirementsvolatility.
•Table summarizes the numerous dimensions of improvement incorporated into the CCPDS-R
project.

60. CCPDS-R technology improvements

61. Reference
•Software Project Management -A Unified Framework
Walker Royce