- This line was added.
- This line was removed.
- Formatting was changed.
Table of Contents
|Table of Contents|
Things We Should Keep Doing
- SPH: Broad regression tests
- SPH: Process: Ticket -> Specs -> Dev -> Test
- SPH: Group/Feature area leads
Things We Should Change
Build-Test System Reliability/Performance Needs Improvement
- SPH: Improve test system reliability. Test systems should fail less than once a month.
- SPH: Get two machines set up to run the tests
- SPH Schedule maintenance work in advance, if possible
- SPH Start system early enough to be monitored
- SPH: Machine should be located on-center, so VPN is not needed
- JJKP: Something keeps causing the script test system to crash when loading results. If this continues, we may need to investigate changing how the reporting code works. (This might be a good idea anyway, it takes a LONG time).
- Bottom line: SCALABILITY. We have almost 11k tests, and both the filtering mechanism and the reporting mechanism aren't behaving well with this size.
- SPH: We should consider farming out GUI tests across more machines to reduce the time to get results. Currently it takes about 5 days to know the state of a build. 3 to run and a 1-2 to analyze. This makes RC cycles take a long time.
- SPH: We were not running certain types of regression tests!!! This can't happen!!
- Build system needs to be on a machine more people can access.
- SPH: When submitting a bug, include scripts or detailed list of procedures, and error messages.
- DJC: When reporting issues:
- If you have a script that shows the issue, always include it even if it seems trivial.
- Avoid "whack a mole" issues, where the developer will fix the presented issue only to see it reopened with a description of "this one is just like the original issue." Reopening once is okay, but if the issue shows signs of being systemic, then change the description to clue the developer in that it is likely systemic, or open a new issue.
- SPH: Critical path issues in ticket system put us at risk and we got lucky
- Lots of tickets waiting for clarification or analysis until last month of 10 month release cycle. We got lucky here for the most part. At least one bug just coulnd't be fixed because of lack of time to solve the problem.
- Lot's of work for SPH and JJKP tracked outside of ticket system. The result is that P2 items in those lists get attention before P1 release items. For example, we sent a lot of time on SBIR support, but didn't start reviewing end user docs until 3 days before scheduled app freeze even though over half the doc was ready weeks before.
- DJC: When partially closing issues:
- Be sure to include text explaining the current status of the issue so that when a developer returns to it, it is clear where things were left. (See for example GMT-2686. I should have noted where things were left, but didn't.)
JJKP: Some Visual Freeze things don't actually need to be in visual freeze.
- Updating destination addresses in GMAT.ini
- Updating TestComplete link tests in GMAT.ini
- Note that if new buttons/links need to be added, these do become Viz Freeze items.
- SPH: Leverage code from heritage systems
- SPH: Consider using more third party libraries like SPICE or Asset Importer to avoid duplication of solved problems
- SPH: Spreading ourselves too thin. Don't do more projects and improvements than we can do well.