Overview, Goals, and Guiding Philosophy

We are in the end-game phase for GMAT v1.0 and our work product is R2013a (Production Release).  To reach this goal, our primary work is to:

  • Explore: Use GMAT and exercise your features
  • Document: Describe final behavior in working specs and User Documentation
  • Test: Complete critical system and beta testing
  • Wrap up: Final inspections and migration

We've found that these activities are not necessarily sequential.  As you test, you discover information that is useful to users and that should be included in specifications.  When you document, you think of areas that could use extra exploratory or beta testing.  You will probably make a few cycles through these activities or treat them as parallel activities.

GMAT has many, many tests.  We are not starting from scratch, we are making a final pass. Beware of letting perfect be the enemy of good in your testing.  It takes a disproportionate amount of time to find the last remaining bugs so balance value of testing with our schedule. Our documentation on the other hand, is not as thorough as our test coverage, and requires a lot of updating and expanding.  Start from existing docs and consolidate and update when possible.
The sections below describe the work flow and deliverables for the Explore, Document, Test areas.  The primary deliverables of this work are:
- High quality features -  Updated working specs - Missing critical tests - User documentation

Explore

The first step is to simply explore the feature and document what you find.  There is a placeholder in the "Explore" section of the feature spec to record a complete list of all known issues during the exploration phase.  We use this list to do a pre-triage/sanity check to determine the state of the feature.
Explore

- Eat our dog food.  Use the feature for "real" analysis or work. - Enter a sample mission, or tests from the test repo through the GUI then create variations and explore the maturity of the implementation. - Look for failing regression tests in nightly reports and check in bugs if they are not already tracked. - Document all known issues at the beginning of the feature finalization in the feature spec and pre-triage with team input

Track

- Check in new bugs in our bug tracker using the policy described here - Use the JIRA ticket named after your feature to track minor to-do and discussions. For example, for Report File use GMT-2653 - For "large" or "significant" issues, create separate tickets

Documentation

The goals of the documentation effort are:

  • To provide working specs that define R2013a behavior
  • To provide high quality end user docs

The basic workflow for documentation is illustrated below. Begin by consolidating existing material from old documents in working specifications and then update the specification for style and completeness.  Completed draft specs are inspected by a developer, tester, and possibly another engineer.  After all issues in the specs are resolved, the material relevant to users is included in end-user documentation and you can use the spec to evaluate existing test coverage.

Old Documentation

Historical and old documenation is currently listed in html help files and in a pdf file that we used to call "Quick Tables".  While this information is not complete, it will provide a good start for completing specs described in the next section.  Here are links to useful, but out of date, functional documentation:

Working Specs

In this section,we'll discuss how to write working specs that will be used to (1) define R2013a behavior, (2) evaluate existing test coverage, and (3)  to write end-user documentation.   We'll discuss each section of the working spec (Requirements, Functional Specifications, and Field Specifications) below.  You'll need the following templates and historical documents for this work.

Requirements and Inspection

Move requirements from the GMAT Requirements spreadsheet show above into the appropriate section in the working spec.  Inspect the requirements asking the following questions:

- Does every requirement map to a completed feature? - Does every feature map to a requirement? - Are the requirements clear? - Are the requirements testable?

Send any issues found when reviewing to the SPH for CCB Analysis.

Test

Wrap Up

Testing

Wrap up tests fall into two areas: save and command mode.  Save mode testing verifies that user configurations are saved to to script files correctly for future sessions.  We test this indirectly by saving test cases and running the saved version against truth data.  Command mode tests verify that Resource configuration works correctly both before and after BeginMissionSequence. Rigorous command mode testing must only be performed on the following objects for R2012a:  Spacecraft  However, for all resources not listed previously, a test must be written to verify a warning is thrown if the user tries to configure the Resource in command mode via a whole-object assignment or by setting a field on the object.

Command Mode Tests

Field Tests

Test that when performing field level assignment in command mode, all fields work correctly.  The following rules apply

  1. You must use non-default values.
  2. You must test each unique configuration.

Create Spacecraft setSet copySat;
GMAT setSet.Attitude = CoordinateSystemFixed;
GMAT setSet.AttitudeCoordinateSystem = 'EarthFixed';
GMAT setSet.EulerAngleSequence = '213';

Create ReportFile TestDataReport TruthDataReport

BeginMissionSequence;

% Apply finite burn to copied spacecraft and report
GMAT copySat.Attitude = CoordinateSystemFixed;
GMAT copySat.AttitudeCoordinateSystem = 'EarthFixed';
GMAT copySat.EulerAngleSequence = '213';

For I = 1:60
    Propagate DefaultProp(copySat) {copySat.ElapsedSecs = 60};
    Report TestDataReport copySat.TAIModJulian copySat.Q1 copySat.Q2 copySat.Q3 copySat.Q4
EndFor
%  Apply finite burn to configured spacecraft and report
For I = 1:60
   Propagate DefaultProp(setSet) {setSet.ElapsedSecs = 60};
   Report TruthDataReport setSet.TAIModJulian setSet.Q1 setSet.Q2 setSet.Q3 setSet.Q4
EndFor

Field Data Type Tests

Test that when a user-data type is consistent with a field type, that you can set the field using the user data type.  The following rules apply

  1. You must use non-default values.
  2. You must test each unique configuration.

Create Spacecraft setSet copySat;
setSat.X = 7120
Create Variable myVar;
myVar    = 7120;
 Create ReportFile TestDataReport TruthDataReport

BeginMissionSequence;

copySat.X = myVar;

For I = 1:60
    Propagate DefaultProp(copySat) {copySat.ElapsedSecs = 60};
    Report TestDataReport copySat.TAIModJulian copySat.Q1 copySat.Q2 copySat.Q3 copySat.Q4
EndFor
%  Apply finite burn to configured spacecraft and report
For I = 1:60
   Propagate DefaultProp(setSet) {setSet.ElapsedSecs = 60};
   Report TruthDataReport setSet.TAIModJulian setSet.Q1 setSet.Q2 setSet.Q3 setSet.Q4
EndFor

Object-toObject Tests

Test that when performing object to object assignment, all fields work correctly.  The following rules apply

  1. You must use non-default values.
  2. You must test each unique configuration.

Here is an examples for Spacecraft where for the CoordinateSystemFixed attitude mode.  Other tests are written for SPICE and Spinner modes.

Create Spacecraft setSet copySat;
GMAT setSet.Attitude = CoordinateSystemFixed;
GMAT setSet.AttitudeCoordinateSystem = 'EarthFixed';
GMAT setSet.EulerAngleSequence = '323';

Create ReportFile 
BeginMissionSequence

% Apply finite burn to copied spacecraft and report
GMAT copySat = setSet;
For I = 1:60
    Propagate DefaultProp(copySat) {copySat.ElapsedSecs = 60};
    Report TestDataReport copySat.TAIModJulian copySat.Q1 copySat.Q2 copySat.Q3 copySat.Q4
EndFor

%  Apply finite burn to configured spacecraft and report
For I = 1:60
   Propagate DefaultProp(setSet) {setSet.ElapsedSecs = 60};
   Report TruthDataReport setSet.TAIModJulian setSet.Q1 setSet.Q2 setSet.Q3 setSet.Q4
EndFor

Validation Tests

Test each field on the object by setting it to a disallowed value and make sure the invalid setting results in an Error. Here is an example for ReportFile.  You will only be able to test a single field in a given test file because as soon as the error is encountered, an error is thrown and execution stops.

Create ReportFile aReport

BeginMissionSequence
aReport.WriteReport = InvalidString;   %  This is an invalid setting
aReport.Precision   = -30;             %  This is an invalid setting

Save Mode

The purpose of Save testing is to verify that when the user saves a script and runs the saved version the results are the same.  This tests that when you input a configuration, the save file contains all of the configuration data.   The GMAT test system has built in functionality to test this and you only have to mark TC files for the test cases you would like to run in Save mode.  Here is a sample TC file (Ballistic_and_Mass_Case01_LEO.tc) configured to run in Save mode.   You simply add the "Save" keyword to the Categories list.  As feature lead it is your responsibility to mark files for full coverage of the feature.  All Resources and Commands must have Save mode tests.

Categories: [Numeric, Dynamics, Save]      
Requirements: [FRR-4,FRR-4.1,FRR-4.3.1,FRR-4.4.1,FRR-4.4.2,FRR-4.4.3,FRR-4.4.4] 
OutputFiles: [      
    {      
        File: Ballistic_and_Mass_Case01_LEO.report,      
        Truth: Ballistic_and_Mass_Case01_LEO.truth,      
        Comparator: PVComparator,      
        Tolerances: 1e-3      
    }      
]      
Bugs: []     

Here is a partial RunDef file configured to run all Save tests on Spacecraft Ballistic and Mass folder.

% This example runs save mode tests on spacecraft ballistic and mass fields
RunDef.Modes       = {'save'};  %  Set mode to save
RunDef.Comparisons = {'truth'}; 
%RunDef.Cases        = {};      %  Must be commented out for "and" filter to work                          
RunDef.Categories   = {'Save'}; % Runs test cases with "Save" in the category field of the TC file
RunDef.Folders      = {'FRR-4_SpacecraftBallisticMass'};
%RunDef.Requirements = {};      % Must be commented out for "and" filter to work    
RunDef.FilterMode   = 'and';    % Telss test system to run tests that meet all criteria

Validation Tests

GUI Delete and Rename

The Resource Tree in the GMAT GUI allows users to delete and rename Resources

The delete and rename functionality must be tested to ensure that:

  1. The resource can be deleted when allowed.
  2. When the resource is referenced by another resource or command, the resource cannot be deleted.
  3. If the resource is renamed, the rename is made on all other resources or commands referencing the renamed resource.

Item 1 is tested by TestComplete. Items 2 and 3 need to be tested manually, for now.

The list of couplings between resources and commands is located in the Couplings sheet in the User Interface Spec.

If your feature can use parameters (using ParameterSelectDialog), you need to check the following behavior:

  • Include the following in the list of child resources:
    • Spacecraft
    • FuelTank
    • Thruster
    • ImpulsiveBurn
    • CoordinateSystem
    • CelestialBody (include in the list, but don't test it, since you can't rename/delete these)
    • Variable
    • Array
    • String
  • Create a new CoordinateSystem and use it as a dependency while referencing a Spacecraft parameter (e.g. oldSat.oldCS.X). GMAT should prevent you from deleting or renaming CoordinateSystem.
  • When referencing a FuelTank or Thruster, you'll need to attach them to a Spacecraft. Test delete and rename for both the Spacecraft and the FuelTank/Thruster.
  • The list of parameters is here: FRR-38 Calculation Parameters

Delete

  1. Update the list (above) of all resources that can be referenced from the parent feature (resource or command). Call these resources "Child A", "Child B", "Child C", etc.
    If the parent can reference a parameter, include these as possible children: Spacecraft, FuelTank, Thruster, ImpulsiveBurn, CoordinateSystem, Variable, Array, String.
    See the yellow note above for details.
  2. Create the setup script:
    1. In GMAT, create the parent resource or command. You may need to create multiple parent resources or commands to cover all the children.
      For each child resource, create the child and configure the parent to reference it. Name the child resources with the prefix "old" (like "oldVar" for a Variable). There should only be ONE reference to any individual child resource in the entire script.
      If the parent can reference the same type of child resource in multiple ways, create multiple instances of the child resource. For example, XYPlot can reference a Variable in both the XVariable and the YVariable fields. Create two Variable resources ("oldVar1" and "oldVar2") and reference one of them from each field.
    2. When all child resources have been configured, save the script as "ResourceName_Delete.script" (for example, "XYPlot_Delete.script". Save it to the input\GuiRenameDelete folder in the test system.
  3. Perform the delete test:
    1. Reopen GMAT and load the setup script ("ResourceName_Delete.script").
    2. One at a time, try deleting all the child resources (with the "old" prefix). For each one, you should get an error with the name of the parent clearly labeled.
    3. Delete the parent.
    4. One at a time, delete all the child resources. All of them should delete successfully.

Rename

  1. Update the list (above) of all resources that can be referenced from the parent feature (resource or command). Call these "Child A", "Child B", "Child C", etc.
    If the parent can reference a parameter, include these as possible children: Spacecraft, FuelTank, Thruster, ImpulsiveBurn, CoordinateSystem. See the note above for details.
  2. Configure the baseline script:
    1. In GMAT, create the parent resource or command. You may need to create multiple parent resources or commands to cover all the children.
    2. For each child resource, create the child and configure the parent to reference it.
    3. Create some output that will visibly change if you change the child resources.
      For example, if the one of the child resources is Thruster, create a Spacecraft that uses the Thruster, then do a burn and report post-burn velocity. When you change a value on the Thruster, you'll see the change in the report.
    4. When all child resources have been configured, save the script as "ResourceName_Rename_Baseline.script" (for example, "XYPlot_Rename_Baseline.script". Save it to the input\GuiRenameDelete folder in the test system.
  3. Configure the results script:
    1. Change the value of one field on each of the child resources so that you will notice a change in the output for each one.
      For example, if the child resource is Thruster, change Thruster.K1 from 300 to 600.
    2. Save the changed script with "Result" in the name. For example, "XYPlot_Rename_Result.script". Save it to the input\GuiRenameDelete folder in the test system.
    3. Run it and observe the output.
  4. Test the renaming:
    1. Load the baseline script and run it.
    2. One by one, Rename the child resources. Make sure each gets renamed in the parent resource or command.
      For CoordinateSystem, also check that the command summary drop-down gets the rename.
    3. Make the same changes interactively that you made in the result script.
    4. Run it and observe the output.
      1. Make sure the results match those from the result script.
      2. Look at the command summary and make sure it gets the new name.
    5. Save the script to a new, temporary file.
    6. Load it and run it. Make sure it has the new names and output matches the previous run.

Requirements

End-User Documentation

Main Tasks

  1. Review all migrated reference pages (DocBook pages)
  2. Migrate new tutorials from training to DocBook

Resources

  • GMAT Documentation Style Guide
  • DocBook reference
  • Docs files are located in <sf>/trunk/doc/help (/src for .xml files).
  • Images are located in help/src/files/images.
  • The Style Guide is based on the Microsoft Manual of Style, 4th edition. A PDF copy is located on the network at: \\mesa-file\595\GMAT\Resources. When in doubt, look at what this book says.
  • For good examples, see Command_For.xml or Resource_Variable.xml.

Review Reference Pages

Review Checklist

  • Is the DocBook page included in the main document?
    • The file should be referenced from help.xml.
    • Check the table of contents in the PDF, under “Part IV. Reference Guide”.
  • Is there an old version of the page that still exists?
    • For example, does the OpenGLPlot page still exist?
    • If it does, remove it from help.xml and delete the .xml file.
  • Have all the parts of the spec been migrated?
    • Should have everything from the “Interface/Functional Spec” section of the spec.
    • User Interface Spec table should be present in the DocBook file (not just a link).
  • Does the DocBook page match the current version of the spec?
    • If the spec has been updated since it was migrated, make sure the DocBook version matches.
  • Has the User Interface Spec table (fields/options table) been migrated accurately?
  • Is there any information missing from the table?
    • Look at the migrated columns only; some columns weren’t supposed to be migrated.
  • Are the images included properly in the GUI section of the DocBook page?
    • Take new screenshots using SnagIt. Save as .png in help\src\files\images folder.
  • Does the DocBook page match the Style Guide?
    • Are all GUI words, resource names, command names tagged with <guilabel>?
    • Is the “See Also” line in the Description section populated? Are the pages linked?
    • Are all references to other pages linked with <xref>?
    • If there are a lot of links to the same page, link the first one only.
  • Does the DocBook page have the appropriate index entries?
    • Add <indexterm> tags just inside the <refentry> tag in the DocBook file.
  • Do code snippets match the Style Guide?
    • No “GMAT” prefix
    • No semicolons
    • Make it as simplified as possible.
  • In code snippets, do the lines extend beyond the gray rectangle?
    • You’ll need to manually shorten them to fix it.
    • If possible, use GMAT’s “...” operator so users can copy/paste.
  • Do code snippets work?
  • Does the text “sound like” the text in the example DocBook pages?
    • The For command page is a good example.
  • Do tables have overlapping text between columns.  Need to add spacing or wrapping if they do.

Procedure

  1. Find migrated DocBook page (name: Resource_name.xml or Command_name.xml)

  2. Open in XMLMind.

  3. Make all changes according to checklist above.

  4. Build the entire doc after changes to make sure it still works and changes can be seen.

  5. Commit new file. Will appear in daily build @ noon.

  6. Send to review partner to mark up (Steve H./Rizwan and Steve C./Joel)

Migrate New Tutorials

List of tutorials:

  • Mars B-Plane (RQ)

  • Finite Burn (DSC)

  • Multiple-Shooting Optimization (SPH)

Notes

  • Do these as time allows after updating all reference pages.
  • Use existing tutorials as model

  • No labels