You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 26 Next »

Table of Contents

How  Lessons Learned are Managed

GMAT lessons learned are things that we did well and should keep doing, or large scale things we should be doing to improve the software or our project.   Lessons learned are discussed by the team and if we decide there is an real issue, required a plan for improvement in required areas.   To make sure we are efficiently handling Lessons learned, here are some high level guidelines. 

What is a Lessons Learned

Lessons learned are issues that cause significant problems or could have caused significant problems, or are issues where we did something that significantly improved the software or our project.   Lessons learned require group discussion and probably a change in team habits, process or strategy.

  • Issue that is putting the project at greater risk than necessary
  • Issue that is causing significant inefficiency
  • Issue that is significantly lowering quality

What is Not a Lessons Learned

A lesson learned is not a minor annoyance, a tweak to an existing process, or something that can be resolved between team members in the every day process of getting work done. Bring these types of issues up at meetings, or work them among the team members involved.

  • Tweak to an existing process
  • Minor annoyance or gripe

Things We Should Keep Doing

  • SPH:  Keep working so well together!
  • SPH: Broad regression tests
  • SPH: Process: Ticket -> Specs -> Dev -> Test -> User Docs
  • SPH: Group/Feature area leads
  • SPH: Focus on quality
  • SPH: CCB prioritization of work items

Things We Should Change

Do Better

Build-Test System Reliability/Performance Needs Improvement

  • SPH: Improve test system reliability.  Test systems should fail less than once a month.
    • SPH: Get two machines set up to run the tests
    • SPH Schedule maintenance work in advance, if possible
    • SPH Start system early enough to be monitored
    • SPH: Machine should be located on-center, so VPN is not needed
    • JJKP: Something keeps causing the script test system to crash when loading results. If this continues, we may need to investigate changing how the reporting code works. (This might be a good idea anyway, it takes a LONG time).
    • Bottom line: SCALABILITY. We have almost 11k tests, and both the filtering mechanism and the reporting mechanism aren't behaving well with this size.
    • SPH:  We should consider farming out GUI tests across more machines to reduce the time to get results.  Currently it takes about 5 days to know the state of a build.  3 to run and a 1-2 to analyze.  This makes RC cycles take a long time.
    • SPH:  We were not running certain types of regression tests!!!  This can't happen!!
    • SPH:  Put in waiver to avoid overnight reboot of machines
  • Build system needs to be on a machine more people can access.

Ticket/Schedule System

  • SPH: When submitting a bug, include scripts or detailed list of procedures, and error messages.
  • DJC: When reporting issues:
    • If you have a script that shows the issue, always include it even if it seems trivial.
    • Avoid "whack a mole" issues, where the developer will fix the presented issue only to see it reopened with a description of "this one is just like the original issue."  Reopening once is okay, but if the issue shows signs of being systemic, then change the description to clue the developer in that it is likely systemic, or open a new issue.
  • SPH: Critical path issues in ticket system put us at risk and we got lucky
    • Lots of tickets waiting for clarification or analysis until last month of 10 month release cycle.  We got lucky here for the most part.  At least one bug just coulnd't be fixed because of lack of time to solve the problem.
    • Lot's of work for SPH and JJKP tracked outside of ticket system.  The result is that P2 items in those lists get attention before P1 release items.  For example, we sent a lot of time on SBIR support, but didn't start reviewing end user docs until 3 days before scheduled app freeze even though over half the doc was ready weeks before.
  • DJC: When partially closing issues:
    • Be sure to include text explaining the current status of the issue so that when a developer returns to it, it is clear where things were left.  (See for example GMT-2686.  I should have noted where things were left, but didn't.)

Miscellaneous

JJKP: Some Visual Freeze things don't actually need to be in visual freeze.

  • Updating destination addresses in GMAT.ini
  • Updating TestComplete link tests in GMAT.ini
  • Note that if new buttons/links need to be added, these do become Viz Freeze items.

DJC: Make sure that the entire team understands the end-of-cycle and start-of-cycle repository procedures. 

  • During RC evaluation, the only repository commits should be those that are part of the release
  • After the release is identified, make sure everyone knows the repositories are frozen – NO COMMITS – until they have been tagged and branched.

Start Doing

  • SPH: Leverage code from heritage systems
  • SPH: Consider using more third party libraries like SPICE or Asset Importer to avoid duplication of solved problems
  • SPH:  Be better at directly responding to GSFC flight projects..  help missions.

Stop Doing

  • SPH: Spreading ourselves too thin.  Don't do more projects and improvements than we can do well.
  • No labels