Tool Qualification FAQ

Document Actions

For information about what ISIS (a tool developer) is doing towards ESCHER compliance, click here.

What are the qualification criteria?
What does "ESCHER qualification" mean?
How can tool Developers provide information to ESCHER ?
Qualification criteria related questions

What are the qualification criteria?

ESCHER conducts surveys and evaluation of embedded systems tools and makes the results available to ESCHER contributors and, in an abridged form, to the public in general. ESCHER has defined certain criteria for evaluating tools. A document outlining the qualification criteria can be downloaded here.

What does "ESCHER qualification" mean?

ESCHER qualification of a tool means that the tool has met, to a certain extent, the criteria defined by ESCHER. While different tools may meet the criteria more than some other tools (e.g., a tool may provide better user support that others in terms of FAQs, help, tutorials, etc.) some minimum requirements must be met:

The tool must use an open source licensing model and the must allow unrestricted use company-wide (in research and/or in product development).
The tool must integrate with at least one of the frameworks listed in the Integratibility criteria.
The tool must provide support for bug reports and user feedback.

How can tool developers provide information to ESCHER?

As a first step, tool developers can fill out a survey. Subsequently, tool developers can update the survey form to provide continual updates (the survey will also evolve). It will be of great help to users if tool developers can also set up web pages which present the information sought by ESCHER in a concise manner.
Additionally, at some time in the future, ESCHER will set up a web based interface for providing information about the tools.
If tool developers want to set up RSS feeds to news items relating to their tools, they can provide the URL to ESCHER and ESCHER will link in their feeds. Many tools for generating RSS feeds are available .

Does ESCHER require the use of specific tools and methodology?

While ESCHER may recommend, purely as suggestions or guidelines, certain tools, systems, platforms, processes or methodology, it does not mandate their use. ESCHER's interest is in assuring that the tools meet the criteria, not in the specific tools, etc.

What is the significance of the licensing model?

Since one of the goals of ESCHER it to promote the development and use of public, open source toos, the licensing model used by a tool is a very important part of ESCHER qualification.

What kind of licensing model and usage policies are required?

A tool must use open souce licensing model and must allow unrestricted use company-wide (in research and/or in product development).

Will ESCHER host information about a tool even if it is not open source or public?

ESCHER will host information about proprietary or restricted tools also. These tools will not be ESCHER qualified.

What does "tool dependencies" mean?

A tool may depend on other tools, software and platforms that may result in the users of the tool incurring additional costs in terms of licensing, hardware, networking, special hardware requirements and even resources required to implement systems using the tool.

Tool developers must fully disclose all such dependencies.

What kind of tool dependencies should be disclosed?

The user should be fully informed of any copyright issues involved in using a tool because of its dependencies. Also, usage of a tool should not change the user's rights to the system that the user is developing.
If a tool uses proprietary software, user should be made aware of any licensing costs associated with that software.
The user should be made aware of any platform compatibility issues.
The number of other tools that will need to be installed and the effort and skills required.
The hardware, networking, etc. configuration and resources needed.

What if a tool is open source but is built upon a commercial tool?

The tool developers must let the users know of this dependency. The tool will be listed in ESCHER repository as an open source/public tool, but with the dependency on the commercial tool noted.

Whether the tool will be considered ESCHER qualified will depend on the degree to which it depends on the commercial tool. For example, if the core functionality of the tool does not require the commercial tool, but only peripheral, "extended", functionality does then it may be listed as ESCHER qualified. This also requires that the core functionality delivers the most value to user.

What about a scenario where an open source tool interfaces to a commercial tool, reads its models and performs analysis, perhaps persisting results back in the commercial tool? In this case the open source tool will be considered ESCHER qualified.

What is the purpose of Functional Integrity criterion?

This criterion seeks to ensure that a tool does what it claims in a verifiable manner. The following artifacts are needed to achieve that:

Functional specifications (or logical design) and any other documents that clearly outline the features and expected behavior of the tool and its interfaces.
Test suites for the tool. The test suites may consist of automated test scripts/executables and manual test scripts.

Test suites for a tool can take many forms such as automated unit and functional testing tools, stress tests, manual scripts (particularly for testing GUI functionality). The important thing is that every feature of a tool can be tested and verified using the test scripts.

Is there a specific methodology or format that should be used for functional specifications?

ESCHER does not require the use of any specific standard. Tool developers are free to choose any formalism(s) that they wish as long as the fomalisms are adequate to represent the functional behavior of the tool. Examples of formalisms are: UML, sequence diagrams, interaction diagrams, use cases, Entity-Relationship (E-R) diagrams, Logical Data Model (LDM), etc.

Are there systems or frameworks for automating test suite execution and generating reports?

Some open source testing tool and systems can be found at http://www.opensourcetesting.org/.

Information about distributed, continuous QA can be found at http://www.cs.umd.edu/projects/skoll/skoll.htm.

How often should the tests be run?

The frequency of testing is largely determined by the development process being used for the tool. At a minimum, the test must be run during the "alpha/beta" testing phase and before the release. Beyond that, if the development process incorporates daily builds or frequent builds, then tests must be run after each build.

How can test results be sent to ESCHER?

The test suite for a tool will typically contain a very large number of test, providing detailed coverage of all the features (low level to high level) of the tool. A tool developer must summarize the results before communicating the results to ESCHER. The summarization should be done keeping the users in mind. While statistics like number of tests run, number of low level features tested, number passed, etc., provide information about general robustness of the tool, they do not give a good sense of robustness of high level functionalities. For example, a tool may provide funtionalities of modeling, model verification, simulation and code generation. In this case a statistic which say that 90% of features passed the test may hide that facts that

100% of low level features of the simulation fucntionality (and hence the entire simulation functionality) passed.
Only 50% of low level features of the code generation functionality passed, making the code generation functionality less usable.

The test results may be communicated to ESCHER in many ways:

Tool developers may send the information in an electonic document. such as a text file, MS Word file, HTML document, XML document, etc.
Additionally, at some time in the future, ESCHER will set up a web based interface for submitting test results.
If tool developers want to set up RSS feeds for test results, they can provide the URL to ESCHER and ESCHER will link in their feeds. Many tools for generating RSS feeds are available.

Which frameworks/systems should tools integrate with?

Tools in the Repository will be required to contain interfaces that conform to the following tool integration frameworks:

OTIF: The Open Tool Integration Framework (OTIF) (http://repo.isis.vanderbilt.edu/tools/get_tool?WOTIF) is software infrastructure for building tool integration solutions. OTIF provides a meta-model driven infrastructure for design tool integration, which facilitates the semantic interoperability across the elements of a tool chain. OTIF is based on open standards such as OMG's MOF and CORBA/IDL and is itself an open standard.
Eclipse: Eclipse (www.eclipse.org) is an open source software development project dedicated to providing a robust, full-featured, commercial-quality, industry platform for the development of highly integrated tools. It provides a focal point for diverse tool builders to ensure the creation of best of breed tools for the Eclipse Platform and provides new channels for open source developers, researchers, academics and educators to participate in the on-going evolution of Eclipse.

The tools must also include test suites, as for Functional Integrity criteria, to determine compliance. The developers should clearly state the specific interfaces that the tool conforms to. A tool may conform to only a subset of the interfaces that OTIF and Eclipse define. The tool will be evaluated for successful implementation of these interfaces only.

Are test suites for integratability needed?

Yes, the developer must

Provide test suites for the interfaces. The test suites may consist of automated test scripts/executables and manual test scripts.
The developer will be responsible for testing the interfaces and generating reports about the test results.

Is testing of integration different from testing for functional integrity?

No -- the same considerations that apply for testing functional integrity also apply for testing integration.

What kind of documentation is needed?

The tools in the Repository must provide the following documentation:

User Manuals
Reference Manuals

The above two artifacts should be made available in paper, electronic and online forms.

The tool developer must prepare the two kinds of documents listed above. Because ESCHER qualification means the tools is open source, the manuals should be developed not only for the use of the tool but also for development using the tool. For exmaple, a timing analysis tool should have the following documents:

User Manual for performing timing analysis (including, perhaps, modeling, simulation, etc.).
Reference Manual for timing analysis
User manual for doing code development using the tools sources (perhaps to integrate the timing analysis into another system)
Reference Manual for the source code, listing every library, class, method, etc.

A user manual describes the use of a tool, what are its features, how to use them. Typically, it is the document that a user first turns to and oftentimes uses as a learning aid.

A Reference Manual, on the other hand is a dry listing of all the functionality and features of a tool, which assume prior knowledge of how to use the tool (perhaps acquired through perusal of the User Manual).

For an example of User Manual vs. Reference Manual, compare the following two:

http://docs.sun.com/db/doc/817-5072

http://docs.sun.com/db/doc/817-5071

Is there a specific format that should be used for User Manuals or Reference Manuals?

ESCHER does not require a specific format for these manuals (the above links are for illustration purposes only, not exemplar), as long as they contain the requisite information.

Is SDK/API documentation needed?

Yes, as noted above, SDK/API documentation is required.

How can better support for users be provided?

A tool should provide artifacts that aid the user, such as:

Download and versioning facilities, such as a CVS Repository for the tool.
Discussion boards/mailing lists with active participation by the tool developers.
Bug Reporting mechanisms, e.g., Bugzilla.The bug reports must be monitored and responded to by the developer.The developers must publish plans for addressing the bugs and incorporate the bug fixes in periodic, scheduled releases and hotfix releases as needed.
Mechanisms for submitting features requests (could be part of the bug reporting system).Feature requests should be handled in the same manner as bug reports, with possibly lower priority.
Tutorials (print media, electronic and/or wed-based)
Training
FAQs
Online Help

Should tool developers set up a download site for their tools?

Yes, tool developers should set up a website with versioning and downloading capabilities.

The versioning mechanism should allow access to earlier versions in addition to the current version. The appropriate release notes, installation instructions, manuals and other documentation must be included with each version download. The developers must manage the versioning and releases in accordance with their development cycles and communicate the schedules to the users via the website.

Are there any guidelines for implementing versioned downloads of tools?

CVS is useful tool for implementing both versioning and download facilities. However, developers must implement Software Configuration Management (SCM), which goes beyondsimple versioning (see, e.g., http://www.sei.cmu.edu/legacy/scm/). There are many public domain tools (http://www.cmcrossroads.com/bradapp/links/scm-links.html#Free_CM_Tools), as well as commercial tools (http://www.cmcrossroads.com/bradapp/links/scm-links.html#Com_CM_Vends) available for SCM.

What kind of discussion boards and mailing lists should tool developers provide?

Developers should setup up web-based discussion forums where users can participate in discussions with other users and developers. Public domain tools like Plone and TikiWiki, among others, can be used to setup the forums. Developers are free to use proprietary tools also. The developers are responsible for monitoring and participating in the forums.

Can discussion boards or mailing lists be hosted by a third party?

There are many vendors that offer hosting of discussion boards and forums that can be found using simple Google searches (http://www.google.com/search?sourceid=navclient&ie=UTF-8&oe=UTF-8&q=discussion+board+hosting or http://www.google.com/search?sourceid=navclient&ie=UTF-8&oe=UTF-8&q=forum+hosting).

ESCHER has not evaluated any of these services and does not make any claims as to the quality, usability or reliabiltiy of these services.

Which tools/systems should be used to support bug/issue reporting (defect tracking)?

Tools such as JIRA and Bugzilla, among others, may be used to implement this functionality. See also http://www.opensourcetesting.org/bugdb.php.

Can the bug/issue reporting system be hosted by a third party?

There are many vendors that offer hosting of bug reporting and issue management systems that can be found using simple Google searches (http://www.google.com/search?sourceid=navclient&ie=UTF-8&oe=UTF-8&q=hosted+bug+reporting or http://www.google.com/search?sourceid=navclient&ie=UTF-8&oe=UTF-8&q=hosted+issue+management).

ESCHER has not evaluated any of these services and does not make any claims as to the quality, usability or reliabiltiy of these services.

What are a tool developer's responsibilities with regards to the bug/issue reports and discussion boards and mailing lists?

The developers must maintain web-based system for reporting bugs and submitting feature requests. The developers must monitor and respond to each submitted bug report and feature request, including indications of plans for addressing the issues (whether a bug will be fixed or not and whether a feature will be added or not; if yes then in which release, etc.).

Is there a specific format to use for Tutorials?

Tutorials should primarily be aimed at getting novices to start using the tool quickly. The format of the tutorial can be to the developers choosing - a document, a web based system, an executable, animation, etc.

What are the different ways tool developers can provide help to the users?

Prepare FAQs for about the tool and make them available as downloadable documents as well as online pages.

In addition to FAQs, discussion forums and other online systems, developers may set up website to answer user's questions and to help him "troubleshoot".

Further, developers may setup RSS feeds on their website to better disseminate information about the tools (ESCHER will hook into the feed). Many tools for generating RSS feeds are available.

What is the significance of Development QA Processes criterion?

Because the quality and usability of software is significantly influenced by the use of a development and QA process (or lack thereof), ESCHER seeks to ensure that tools that are ESCHER qualified have used such processes fortheir development. This willgoa long way towards a tool's maturity and acceptability in the industry.

The development processes used for tool development must be transparent and must employ some commonly used QA procedures and mechanisms, such as:

A well defined process for requirements gathering, design, development, testing and stabilization of the tool, with specific milestones for each stage.
Clear separation of conceptual, logical and physical design.
Coding standards.
Code reviews.
Daily builds.
Testing and stabilization.

The developers must document the process used for the development life-cycle. The document should describe all aspects of a development cycle listed in the criterion and any additional information.

Which development process or methodology should be used?

ESCHER does not recommend any specific methodology, process or framework. Developers are free to choose the one that best fits their team and requirements. However, the chosen methodology, process or framework must include the items listed in the criterion.

An example of the use of a process and its documentation can be found at http://ptolemy.eecs.berkeley.edu/ptolemyII/.

Should one use any specific coding standards?

ESCHER does not recommend any specific coding standard. We do, however, strongly recommend using some coding standard to ensure quality in the tool.

What are daily builds?

For a large, complex sofwtare with many components, daily builds are perhaps the most important quality assurance mechanism.

In a nutshell, daily builds consist of building the complete tool (re-compling and linking all the components), deploying and then testing the tool. It ensures that the components remain compatible with each other and can be integrated together, even as the individual developers work on adding features or fixing bugs. Lack of daily builds in a developmentprocess introduces large cost and quality risks into integration, testing and stabilization of the software.

Daily builds have also been called the "heartbeat" of the software -- a successful daily build (compile, link, delpoy, test) means that the software is alive and well.

Must tool developers implement a development process that includes daily builds?

Yes, ESCHER strongly recommends including daily builds in the development process.

Are there tools one can use to implement daily builds?

The development tool/platform/environment that one uses may come with facilities to automate builds.

There are many vendors that offer tools for automating builds that can be found using simple Google searches (http://www.google.com/search?sourceid=navclient&ie=UTF-8&oe=UTF-8&q=software+build+tools).

ESCHER has not evaluated any of these tools and does not make any claims as to the quality, usability or reliabiltiy of these tools .

ESCHER Research Institute

Sections

Personal tools

Navigation

Tool Qualification FAQ

Document Actions