Commit 1a137d37 authored by Gilles Mouchard's avatar Gilles Mouchard
Browse files

Updated "Open source PKM server-side software" and "Open source client-side software".

parent 04cce63a
Pipeline #18049 passed with stages
in 9 minutes and 11 seconds
......@@ -2,7 +2,7 @@
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="date" content='12/10/21'>
<meta name="date" content='12/14/21'>
<title>D1.5 Open source PKM server-side software</title>
<link rel="stylesheet" href='../style/style.css' type="text/css">
</head>
......@@ -18,11 +18,11 @@
<div id="content">
<h1 id="introduction">1 Introduction</h1>
<h2 id="purpose">1.1 Purpose <a id="1"></a></h2>
<p>The PKM (Persistent Knowledge Monitor) is a living repository for every project, relying on a database management system (MongoDB) and the local server storage for the Git repositories, where different representations are kept (including source code, design models, specifications, requirements, etc.) as well as their links in a traceability matrix. The PKM can be queried and enriched by the actors involved in the project, in order to maintain consistency and keep the most updated and precise information about the project.</p>
<p>The PKM (Persistent Knowledge Monitor) is a living repository for every software project, relying on a database management system (MongoDB), where different representations are kept (including source code, design models, specifications, requirements, etc.) as well as their links in a traceability matrix. The PKM uses the local storage of the server to synchronize with external Git repositories. The PKM can be queried and enriched by the actors involved in the project, in order to maintain consistency and keep the most updated and precise information about the project.</p>
<p>This document, which presents the architecture of the PKM server (i.e. the backend) and the format of the different sorts of knowledge that need to be maintained together with the code in its different forms (source, binary, byte code), is intended for:</p>
<ul>
<li>A user or a system administrator willing to install the PKM,</li>
<li>A Developer willing to maintain and extend the PKM source code.</li>
<li>A developer willing to maintain and extend the PKM source code.</li>
</ul>
<p>The purpose of this document is not about the interactions of clients with the PKM which is the purpose of sibling document titled “Open source client-side software”.</p>
<p>The source code of the PKM is available at <a href="https://gitlab.ow2.org/decoder/pkm-api">https://gitlab.ow2.org/decoder/pkm-api</a>.</p>
......@@ -31,9 +31,9 @@
<p>Copyright © 2020-2021 Capgemini Group, Commissariat à l’énergie atomique et aux énergies alternatives, OW2, Sysgo AG, Technikon, Tree Technology, Universitat Politècnica de València.</p>
<p>The PKM server is licensed under <a href="https://gitlab.ow2.org/decoder/pkm-api/-/raw/master/AGPL-3.0.txt">GNU Affero General Public License version 3</a>.</p>
<p>The SDKs for the clients of PKM and parsers are licensed under <a href="https://gitlab.ow2.org/decoder/pkm-api/-/raw/master/Apache-2.0.txt">Apache License version 2.0</a>.</p>
<p>The parsers and tools are licensed under there respective licenses.</p>
<p>The parsers and tools are licensed under their respective licenses.</p>
<h2 id="choosing-a-database">1.3 Choosing a database <a id="3"></a></h2>
<p>The DECODER project partners have sought for a valuable and adequate implementation of the PKM by examining several open-source document-oriented DBs, OrientDB and CouchDB and NoSQL, and the current preferred choice has led us to go for MongoDB V3.4. Indeed, the data used in the project is structured into complex JSON documents, grouped into Collections. Documents are linked together by means of internal fields such as file names, function names or identifiers. The PKM has been tested with MongoDB 3.4 and later (up to version 4.4 at the time of writing this document).</p>
<p>The DECODER project partners have sought for a valuable and adequate implementation of the PKM by examining several open-source document-oriented DBs, OrientDB and CouchDB and NoSQL, and the current preferred choice has led us to go for MongoDB V3.4. Indeed, the data used in the project is structured into complex JSON documents, grouped into Collections. Documents are linked together by means of internal fields such as file names, function names or identifiers. The PKM has been tested with MongoDB 3.4 and later (up to version 4.4 at the time of writing this document). Note that after MongoDB 3.4, MongoDB license changed from AGPL (GNU Affero General Public License) open source license to SSPL (Server Side Public License). The purpose of the new license is to prohibit the sale of raw MongoDB storage. Although OSI (Open Source Initiative) does not consider the new license open source, since the PKM purpose is not selling raw MongoDB storage, this change should not affect business and merchantability of the PKM.</p>
<p>However, the partners have had to deal with some technical limitations of MongoDB:</p>
<ul>
<li>Database name: on Linux, database name must not contain <code>'/'</code> (slash), <code>'\'</code> (backslash), <code>'.'</code> (dot), <code>'"'</code> (double quote), or <code>'$'</code> (dollar) characters. This has affected the naming convention of PKM projects because a PKM project is a MongoDB database.</li>
......@@ -44,7 +44,7 @@
<p>Chapter 2, which is intended for users and system administrators, presents the building, configuration, and installation process of the PKM and parsers. Chapters 3, 4, and 5, which are intended for developers, present the PKM implementation. Chapter 6 concludes this document. The appendix talks about the Javascript SDK of the PKM server, the generation flow of the REST servers and SDKs for the clients, and the test procedure that was implemented to automate the test of the PKM server and more generally the DECODER Project tool-chain.</p>
</div>
<div id="footer">
<span><a href="../index.html">D1.5 Open source PKM server-side software</a> - 12/10/21</span>
<span><a href="../index.html">D1.5 Open source PKM server-side software</a> - 12/14/21</span>
</div>
</body>
</html>
......@@ -2,13 +2,14 @@
## 1.1 Purpose <a id="1"></a>
The PKM (Persistent Knowledge Monitor) is a living repository for every project, relying on a database management system (MongoDB) and the local server storage for the Git repositories, where different representations are kept (including source code, design models, specifications, requirements, etc.) as well as their links in a traceability matrix.
The PKM (Persistent Knowledge Monitor) is a living repository for every software project, relying on a database management system (MongoDB), where different representations are kept (including source code, design models, specifications, requirements, etc.) as well as their links in a traceability matrix.
The PKM uses the local storage of the server to synchronize with external Git repositories.
The PKM can be queried and enriched by the actors involved in the project, in order to maintain consistency and keep the most updated and precise information about the project.
This document, which presents the architecture of the PKM server (i.e. the backend) and the format of the different sorts of knowledge that need to be maintained together with the code in its different forms (source, binary, byte code), is intended for:
* A user or a system administrator willing to install the PKM,
* A Developer willing to maintain and extend the PKM source code.
* A developer willing to maintain and extend the PKM source code.
The purpose of this document is not about the interactions of clients with the PKM which is the purpose of sibling document titled "Open source client-side software".
......@@ -24,13 +25,15 @@ The PKM server is licensed under [GNU Affero General Public License version 3](h
The SDKs for the clients of PKM and parsers are licensed under [Apache License version 2.0](https://gitlab.ow2.org/decoder/pkm-api/-/raw/master/Apache-2.0.txt).
The parsers and tools are licensed under there respective licenses.
The parsers and tools are licensed under their respective licenses.
## 1.3 Choosing a database <a id="3"></a>
The DECODER project partners have sought for a valuable and adequate implementation of the PKM by examining several open-source document-oriented DBs, OrientDB and CouchDB and NoSQL, and the current preferred choice has led us to go for MongoDB V3.4.
Indeed, the data used in the project is structured into complex JSON documents, grouped into Collections. Documents are linked together by means of internal fields such as file names, function names or identifiers.
The PKM has been tested with MongoDB 3.4 and later (up to version 4.4 at the time of writing this document).
Note that after MongoDB 3.4, MongoDB license changed from AGPL (GNU Affero General Public License) open source license to SSPL (Server Side Public License). The purpose of the new license is to prohibit the sale of raw MongoDB storage.
Although OSI (Open Source Initiative) does not consider the new license open source, since the PKM purpose is not selling raw MongoDB storage, this change should not affect business and merchantability of the PKM.
However, the partners have had to deal with some technical limitations of MongoDB:
......
......@@ -2,7 +2,7 @@
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="date" content='12/10/21'>
<meta name="date" content='12/14/21'>
<title>D1.5 Open source PKM server-side software</title>
<link rel="stylesheet" href='../style/style.css' type="text/css">
</head>
......@@ -37,8 +37,8 @@
&quot;file_types&quot;: […]
}</code></pre>
<p>The JSON schema of the configuration file is available on the <a href="https://gitlab.ow2.org/decoder/pkm-api/-/blob/master/api/pkm-config-schema.json">pkm-api gitlab repository</a>.</p>
<h2 id="easy-installation-no-building-required">2.2 Easy installation (no building required)<a id="2"></a></h2>
<p>A program that provides a command line interface and a control center for managing the services (download, install, update, start …) of DECODER EU Project Tool-chain is available at <a href="https://gitlab.ow2.org/decoder/docker-deployment">https://gitlab.ow2.org/decoder/docker-deployment</a>. This program is based on <a href="https://www.docker.com/">docker</a> and <a href="https://docs.docker.com/compose">docker-compose.</a></p>
<h2 id="easy-installation-no-build-required">2.2 Easy installation (no build required)<a id="2"></a></h2>
<p>A program that provides a command line interface and a control center for managing the services (download, install, update, start …) of DECODER EU Project Tool-chain, as a local instance without remote access, is available at <a href="https://gitlab.ow2.org/decoder/docker-deployment">https://gitlab.ow2.org/decoder/docker-deployment</a>. This program is based on <a href="https://www.docker.com/">docker</a> and <a href="https://docs.docker.com/compose">docker-compose.</a></p>
<p>To run the program, do the following at the command prompt:</p>
<pre><code>$ ./app/bin/decoder-eu</code></pre>
<p>On the first use, you should <code>install</code> then <code>start</code> the toolchain, see Figure 1 below, which shows the control center user’s interface:<a id="figure1"></a></p>
......@@ -65,7 +65,7 @@
<li>zenity (optional)</li>
</ul>
<p><strong>Command line interface</strong></p>
<pre><code>Usage: decoder-eu [&lt;options&gt;] [&lt;command&gt;]
<pre><code>Usage: decoder-eu [&lt;options&gt;] [&lt;commands&gt;]
Options:
--version
......@@ -161,7 +161,7 @@ Commands:
<pre><code>$ docker-compose up</code></pre>
</div>
<div id="footer">
<span><a href="../index.html">D1.5 Open source PKM server-side software</a> - 12/10/21</span>
<span><a href="../index.html">D1.5 Open source PKM server-side software</a> - 12/14/21</span>
</div>
</body>
</html>
......@@ -26,9 +26,9 @@ The default configuration file, which content is outlined below, is good for mos
The JSON schema of the configuration file is available on the [pkm-api gitlab repository](https://gitlab.ow2.org/decoder/pkm-api/-/blob/master/api/pkm-config-schema.json).
## 2.2 Easy installation (no building required)<a id="2"></a>
## 2.2 Easy installation (no build required)<a id="2"></a>
A program that provides a command line interface and a control center for managing the services (download, install, update, start …) of DECODER EU Project Tool-chain is available at [https://gitlab.ow2.org/decoder/docker-deployment](https://gitlab.ow2.org/decoder/docker-deployment).
A program that provides a command line interface and a control center for managing the services (download, install, update, start …) of DECODER EU Project Tool-chain, as a local instance without remote access, is available at [https://gitlab.ow2.org/decoder/docker-deployment](https://gitlab.ow2.org/decoder/docker-deployment).
This program is based on [docker](https://www.docker.com/) and [docker-compose.](https://docs.docker.com/compose)
To run the program, do the following at the command prompt:
......@@ -60,7 +60,7 @@ As soon as the graphical user interface has started in your default web browser,
**Command line interface**
Usage: decoder-eu [<options>] [<command>]
Usage: decoder-eu [<options>] [<commands>]
Options:
--version
......
......@@ -2,7 +2,7 @@
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="date" content='12/10/21'>
<meta name="date" content='12/14/21'>
<title>D1.5 Open source PKM server-side software</title>
<link rel="stylesheet" href='../style/style.css' type="text/css">
</head>
......@@ -27,17 +27,17 @@
<figure>
<img src="pkm-server-software-architecture.png" alt="Figure 3: PKM server software architecture" /><figcaption aria-hidden="true">Figure 3: PKM server software architecture</figcaption>
</figure>
<p>The bottom layer encompasses the underlying Node.js Javascript run-time, the MongoDB Node.js driver, the MongoDB database server, the local file system, and Git. On top of this, the <em>PKM util</em> layer is a kind of hardware abstraction layer that abstracts file system I/O and simplifies the process of tools execution. The <em>PKM core</em> provides Class <code>PKM</code>, which implements the <code>PKM</code> and is responsible of database accesses using the <em>MongoDB Node.js driver</em>. <em>PKM util</em> and <em>PKM core</em> constitute a Javascript SDK for the PKM which purpose for third party contributors is mostly for extending the PKM. On the highest level, there are two kinds of front-ends:</p>
<p>The bottom layer encompasses the underlying Node.js Javascript run-time, the MongoDB Node.js driver, the MongoDB database server, the local file system, and Git. On top of this, the <em>PKM util</em> layer is a kind of hardware abstraction layer that abstracts file system I/O and simplifies the process of tools execution. The <em>PKM core</em> provides Class <code>PKM</code>, which implements the <code>PKM</code> and is responsible of database accesses using the <em>MongoDB Node.js driver</em>. <em>PKM util</em> and <em>PKM core</em> constitute a Javascript SDK for the PKM whose purpose is mostly to allow third party contributors to extend the PKM. On the highest level, there are two kinds of front-ends:</p>
<ul>
<li>the user’s console where the user (or scripts) can interact with the PKM,</li>
<li>and the user’s interface &amp; tools, which can communicate with the PKM.</li>
</ul>
<p>These front-ends use two different Application Programming Interfaces:</p>
<ul>
<li>The <em>PKM cli</em> provides a command line interface for the PKM which purposes are essentially doing administrative tasks and debugging the PKM,</li>
<li>The <em>PKM RESTful HTTP server</em> provides a remote interface over HTTP following the OpenAPI 3 standard which purpose is fundamentally allowing connecting front-ends, e.g. running within a web browser, or third party tools running on remote servers.</li>
<li>The <em>PKM cli</em> provides a command line interface for the PKM whose purposes are essentially doing administrative tasks and debugging the PKM,</li>
<li>The <em>PKM RESTful HTTP server</em> provides a remote interface over HTTP following the OpenAPI 3 standard whose purpose is fundamentally allowing connecting front-ends, e.g. running within a web browser, or third party tools running on remote servers.</li>
</ul>
<p>The PKM and parsers source code is available at <a href="https://gitlab.ow2.org/decoder/pkm-api/-/blob/master">https://gitlab.ow2.org/decoder/pkm-api/-/blob/master</a>. Because the choosen language for developing the PKM server is javascript, which is a dynamically typed language, most of the source code has been annotated using the <a href="https://jsdoc.app"><code>jsdoc</code></a> markup language (see Appendix A.1) to document the parameters and the return value of each function.</p>
<p>The PKM and parsers source code is available at <a href="https://gitlab.ow2.org/decoder/pkm-api/-/blob/master">https://gitlab.ow2.org/decoder/pkm-api/-/blob/master</a>. Because the choosen language for developing the PKM server is Javascript, which is a dynamically typed language, most of the source code has been annotated using the <a href="https://jsdoc.app"><code>jsdoc</code></a> markup language (see Appendix A.1) to document the parameters and the return value of each function.</p>
<p>The following subsections presents the MongoDB database and the PKM Application Programming Interfaces. The Git subsystem, which has received a lot of attention, as it is essential software for developers, has Chapter 4 dedicated to it.</p>
<h2 id="mongodb-database">3.2 MongoDB database <a id="2"></a></h2>
<p>The MongoDB server can host several databases at a time, and each database can have several users and roles. One special database, named <code>'admin'</code>, defines the MongoDB built-in roles, such as <code>'root'</code>. A PKM administrator has role <code>'root'</code> in database <code>'admin'</code>.</p>
......@@ -46,7 +46,7 @@
<figure>
<img src="pkm-collections.png" alt="Figure 4: PKM MongoDB collections" /><figcaption aria-hidden="true">Figure 4: PKM MongoDB collections</figcaption>
</figure>
<p>The <code>Project</code> collection hosts the project metadata (name, members). Each project have a collection for Git working trees metadata, which the PKM server updates after running Git commands. Some collections track the tool executions (invocations and logs). The tool specifications enable GUI front-ends to create well-formed tool invocations (for the Process Engine). The methodology have a collection for the status of each phase of the methodology. The programming artefacts have both collections for files (both executable binaries and source codes), for compile commands (with compilation flags and options), and (after parsing) for source code Abstract Syntax Trees (ASTs), comments, and annotations. The models have also both collections for the UML files, and (after parsing) for UML class diagrams and state machines. The documentation have a collection for the documentation files (e.g. .docx files), for the Abstract Semi-Formal Models (ASFMs), which Doc to ASFM can generate from .docx files, and for the Graphical documentation written in the Graphical Specification Language (GSL). Links discovering tool (Semantic parsing) populates the 2D traceability matrix, while Semantic Role Labeling (SRL) and Named Entity Recognition (NER) tools extract information and synthesize Annotations in a dedicated collection. The TESTAR tool (automated GUI testing) have specialized collections for settings, models, and results. Finally, there are collections for Common Vulnerabilities and Exposures, and the reviews. Note that from the PKM REST API point of view the tool specifications, the TESTAR settings, and the methodology status are also properties of the project, even if in reality they are stored in separate collections. The PKM populates, at project creation, the initial methodology status and a predefined set of tool specifications.</p>
<p>The <code>Project</code> collection hosts the project metadata (name, members). Each project has a collection for Git working trees metadata, which the PKM server updates after running Git commands. Some collections track the tool executions (invocations and logs). The tool specifications enable GUI front-ends to create well-formed tool invocations (for the Process Engine). The methodology have a collection for the status of each phase of the methodology. The programming artefacts have both collections for files (both executable binaries and source codes), for compile commands (with compilation flags and options), and (after parsing) for source code Abstract Syntax Trees (ASTs), comments, and annotations. The models have also collections for both the UML files, and (after parsing) for UML class diagrams and state machines. The documentation have a collection for the documentation files (e.g. .docx files), for the Abstract Semi-Formal Models (ASFMs), which Doc to ASFM can generate from .docx files, and for the Graphical documentation written in the Graphical Specification Language (GSL). Links discovering tool (Semantic parsing) populates the 2D traceability matrix, while Semantic Role Labeling (SRL) and Named Entity Recognition (NER) tools extract information and synthesize Annotations in a dedicated collection. The TESTAR tool (automated GUI testing) has specialized collections for settings, models, and results. Finally, there are collections for Common Vulnerabilities and Exposures, and the reviews. Note that from the PKM REST API point of view the tool specifications, the TESTAR settings, and the methodology status are also properties of the project, even if in reality they are stored in separate collections. The PKM populates, at project creation, the initial methodology status and a predefined set of tool specifications.</p>
<p>The following subsections detail the collections in a PKM project, which table below summarizes:</p>
<table>
<thead>
......@@ -312,12 +312,12 @@
<h3 id="files">3.2.2 Files <a id="2.2"></a></h3>
<p>The PKM organizes files per collection depending on the file type. The collections for the PKM files in MongoDB are:</p>
<ul>
<li><code>'RawSourcecode'</code> for the source codes</li>
<li><code>'RawSourcecode'</code> for the source code</li>
<li><code>'RawUML'</code> for the UML models</li>
<li><code>'RawDocumentation'</code> for the documentation (e.g. Microsoft Office .docx files)</li>
<li><code>'RawBinaries'</code> for the executable binaries (e.g. Linux ELF files)</li>
</ul>
<p>From the PKM REST API point of view, these file collections are shown either as four independent endpoints, or as a one unified endpoint for any file. For the later endpoint, the PKM guesses the actual underlying collection by applying some rules about the filename extension and the MIME type. The PKM configuration file <code>pkm_config.json</code> allows extending these rules throught the <code>file_types</code> property, see Section 2.1.</p>
<p>From the PKM REST API point of view, these file collections are shown either as four independent endpoints, or as a one unified endpoint for any file. For the later endpoint, the PKM guesses the actual underlying collection by applying some rules about the filename extension and the MIME type. The PKM configuration file <code>pkm_config.json</code> allows extending these rules through the <code>file_types</code> property, see Section 2.1.</p>
<p>The schema below for the PKM files in MongoDB can be downloaded from the <a href="https://gitlab.ow2.org/decoder/pkm-api/-/blob/master/api/pkm-db-file-schema.json">pkm-api gitlab repository</a>:</p>
<pre><code>{
&quot;$schema&quot;: &quot;http://json-schema.org/draft-07/schema#&quot;,
......@@ -376,7 +376,7 @@
<li><code>'sourcecodeJava'</code> for Java</li>
<li><code>'sourcecodeCPP'</code> for C++</li>
</ul>
<p>The <code>sourcecodeC</code> collection schema is derived from the Frama-C CIL AST. Due to its extension, here we only show the main structure. The complete schema can be downloaded from the <a href="https://gitlab.ow2.org/decoder/pkm-api/-/blob/master/api/pkm-c-source-code-schema.json">pkm-api gitlab repository</a>. The main structure is the following:</p>
<p>The <code>sourcecodeC</code> collection schema is derived from the Frama-C CIL AST. Due to its size, here we only show the main structure. The complete schema can be downloaded from the <a href="https://gitlab.ow2.org/decoder/pkm-api/-/blob/master/api/pkm-c-source-code-schema.json">pkm-api gitlab repository</a>. The main structure is the following:</p>
<pre><code>{
&quot;$schema&quot;: &quot;http://json-schema.org/draft-07/schema#&quot;,
&quot;$id&quot;: &quot;https://gitlab.ow2.org/decoder/pkm-api/-/tree/master/api/pkm-c-source-code-schema.json&quot;,
......@@ -454,7 +454,7 @@
//Contains all the input definitions
}
}</code></pre>
<p>The <code>sourcecodeCPP</code> collection schema is derived from the Clang AST. Due to its extension, here we only show the main structure. The complete schema can be downloaded from the <a href="https://gitlab.ow2.org/decoder/pkm-api/-/blob/master/api/pkm-cpp-source-code-schema.json">pkm-api gitlab repository</a>. The main structure is the following:</p>
<p>The <code>sourcecodeCPP</code> collection schema is derived from the Clang AST. Again, due to its size, here we only show the main structure. The complete schema can be downloaded from the <a href="https://gitlab.ow2.org/decoder/pkm-api/-/blob/master/api/pkm-cpp-source-code-schema.json">pkm-api gitlab repository</a>. The main structure is the following:</p>
<pre><code>{
&quot;$schema&quot;: &quot;http://json-schema.org/draft-07/schema#&quot;,
&quot;$id&quot;: &quot;https://gitlab.ow2.org/decoder/pkm-api/-/tree/master/api/pkm-cpp-source-code-schema.json&quot;,
......@@ -485,7 +485,7 @@
//Contains all the input definitions
}
}</code></pre>
<p>The <code>sourcecodeJava</code> collection schema is derived from the AST of the Eclipse compiler. Due to its extension, here we only show the main structure. The complete schema can be downloaded from the <a href="https://gitlab.ow2.org/decoder/pkm-api/-/blob/master/api/pkm-java-source-code-schema.json">pkm-api gitlab repository</a>. The main structure is the following:</p>
<p>The <code>sourcecodeJava</code> collection schema is derived from the AST of the Eclipse compiler. Again, due to its size, here we only show the main structure. The complete schema can be downloaded from the <a href="https://gitlab.ow2.org/decoder/pkm-api/-/blob/master/api/pkm-java-source-code-schema.json">pkm-api gitlab repository</a>. The main structure is the following:</p>
<pre><code>{
&quot;$schema&quot;: &quot;http://json-schema.org/draft-07/schema#&quot;,
&quot;$id&quot;: &quot;https://gitlab.ow2.org/decoder/pkm-api/-/tree/master/api/pkm-java-source-code-schema.json&quot;,
......@@ -1807,7 +1807,7 @@
//Contains all the input definitions
}
}</code></pre>
<p>A methodology has several phases, which status consists in the status of each phases of the methodology. The complete schema for the phase status of the methodology can be downloaded from the <a href="https://gitlab.ow2.org/decoder/pkm-api/-/blob/master/api/pkm-methodology-status-schema.json">pkm-api gitlab repository</a>. The main structure of the schema is the following:</p>
<p>A methodology has several phases, and its global status consists in the status of each phase of the methodology. The complete schema for the phase status of the methodology can be downloaded from the <a href="https://gitlab.ow2.org/decoder/pkm-api/-/blob/master/api/pkm-methodology-status-schema.json">pkm-api gitlab repository</a>. The main structure of the schema is the following:</p>
<pre><code>{
&quot;$schema&quot;: &quot;http://json-schema.org/draft-07/schema#&quot;,
&quot;$id&quot;: &quot;https://gitlab.ow2.org/decoder/pkm-api/-/tree/master/api/pkm-methodology-status-schema.json&quot;,
......@@ -1877,7 +1877,7 @@
<li>Functions <code>frama_c</code>, <code>frama_clang</code>, <code>doc_to_asfm</code>, <code>excavator</code> and <code>git</code> use the command line interface of the respective tools that run within a working directory managed with Class <code>FileSystem</code>.</li>
</ul>
<p>The source code of each part of the Javascript SDK is on the pkm-api gitlab repository respectively in directory <a href="https://gitlab.ow2.org/decoder/pkm-api/-/tree/master/core"><code>core</code></a> and <a href="https://gitlab.ow2.org/decoder/pkm-api/-/tree/master/util"><code>util</code></a>.</p>
<p>Most of functions and methods use an asynchronous control flow pattern using javascript Promises. Concretely that means that every API functions and methods are non-blocking and returns immediately an instance of Class <code>Promise</code>, before actual work is finished. The javascript runtime triggers callbacks once the promise resolves. Code below shows the caveat when coding with promises:</p>
<p>Most of functions and methods use an asynchronous control flow pattern using Javascript Promises. Concretely that means that all API functions and methods are non-blocking and returns immediately an instance of Class <code>Promise</code>, before actual work is finished. The Javascript runtime triggers callbacks once the promise resolves. Code below shows the caveat when coding with promises:</p>
<pre><code>var expected_result;
pkm.foobar().then((result) =&gt;
{
......@@ -1893,7 +1893,7 @@ pkm.foobar().then((result) =&gt;
console.log(&#39;expected_result:&#39;, expected_result); // ⚠️ content of expected_result is likely to be undefined
// because not yet assigned
...</code></pre>
<p>Developers not used to using javascript promises who would like to extend the PKM server can find more information about javascript promises by consulting the resources below:</p>
<p>Developers not used to using Javascript promises who would like to extend the PKM server can find more information about Javascript promises by consulting the resources below:</p>
<ul>
<li><a href="https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise">MDN Web Docs: Promise</a></li>
<li><a href="https://www.javascripttutorial.net/es6/javascript-promises">Javascript Tutorial: The Definitive Guide to the JavaScript Promises</a></li>
......@@ -1989,13 +1989,13 @@ PKM.logout(key).then((pkm) =&gt;
<li><code>update_documents_spanned(dbName, collection_names, documents, dispatch, signature)</code></li>
<li><code>insert_update_documents_spanned(dbName, collection_names, documents, update, dispatch, signature)</code></li>
</ul>
<p>There are also some specialized methods but still quite generic methods named <code>insert_update_files</code>, <code>insert_update_files_spanned</code> and <code>delete_files_spanned</code> to manage files:</p>
<p>There are also some specialized but still quite generic methods named <code>insert_update_files</code>, <code>insert_update_files_spanned</code> and <code>delete_files_spanned</code> to manage files:</p>
<ul>
<li><code>insert_update_files(dbName, collection_name, files, update, options = {})</code></li>
<li><code>insert_update_files_spanned(dbName, collection_names, files, update, dispatch, options = {})</code></li>
<li><code>delete_files_spanned(dbName, collection_names, query = {})</code></li>
</ul>
<p>Class <code>PKM</code> provides developer with many specialized methods that rely on these generic methods to manage all type of documents and files that PKM supports.</p>
<p>Class <code>PKM</code> provides developer with many specialized methods that rely on these generic methods to manage all types of documents and files that PKM supports.</p>
<h4 id="document-querying">3.3.1.3 Document querying <a id="3.1.3"></a></h4>
<p>Class <code>PKM</code> provides developer with some generic methods named <code>get_documents</code> and <code>get_documents_spanned</code> to query a document in collections:</p>
<ul>
......@@ -2063,7 +2063,7 @@ PKM.logout(key).then((pkm) =&gt;
<p>The <em>PKM REST API</em> provides access to the PKM over HTTP/HTTPS. The PKM has an <a href="https://www.openapis.org">OpenAPI</a> 3 specification available at <a href="https://gitlab.ow2.org/decoder/pkm-api/-/blob/master/api/pkm-openapi.yaml">https://gitlab.ow2.org/decoder/pkm-api/-/blob/master/api/pkm-openapi.yaml</a>, which enables to automatically generate the SDK for many programming languages, see Appendix A.2. Appendix A.2 contains detailed explanations about implementation design of the REST server, which provides front-ends and third party tools developers with the PKM REST API.</p>
</div>
<div id="footer">
<span><a href="../index.html">D1.5 Open source PKM server-side software</a> - 12/10/21</span>
<span><a href="../index.html">D1.5 Open source PKM server-side software</a> - 12/14/21</span>
</div>
</body>
</html>
......@@ -16,7 +16,7 @@ Figure 3 below shows the layered software architecture of the PKM server and its
The bottom layer encompasses the underlying Node.js Javascript run-time, the MongoDB Node.js driver, the MongoDB database server, the local file system, and Git.
On top of this, the *PKM util* layer is a kind of hardware abstraction layer that abstracts file system I/O and simplifies the process of tools execution.
The *PKM core* provides Class `PKM`, which implements the `PKM` and is responsible of database accesses using the *MongoDB Node.js driver*.
*PKM util* and *PKM core* constitute a Javascript SDK for the PKM which purpose for third party contributors is mostly for extending the PKM.
*PKM util* and *PKM core* constitute a Javascript SDK for the PKM whose purpose is mostly to allow third party contributors to extend the PKM.
On the highest level, there are two kinds of front-ends:
* the user's console where the user (or scripts) can interact with the PKM,
......@@ -24,11 +24,11 @@ On the highest level, there are two kinds of front-ends:
These front-ends use two different Application Programming Interfaces:
* The *PKM cli* provides a command line interface for the PKM which purposes are essentially doing administrative tasks and debugging the PKM,
* The *PKM RESTful HTTP server* provides a remote interface over HTTP following the OpenAPI 3 standard which purpose is fundamentally allowing connecting front-ends, e.g. running within a web browser, or third party tools running on remote servers.
* The *PKM cli* provides a command line interface for the PKM whose purposes are essentially doing administrative tasks and debugging the PKM,
* The *PKM RESTful HTTP server* provides a remote interface over HTTP following the OpenAPI 3 standard whose purpose is fundamentally allowing connecting front-ends, e.g. running within a web browser, or third party tools running on remote servers.
The PKM and parsers source code is available at [https://gitlab.ow2.org/decoder/pkm-api/-/blob/master](https://gitlab.ow2.org/decoder/pkm-api/-/blob/master).
Because the choosen language for developing the PKM server is javascript, which is a dynamically typed language, most of the source code has been annotated using the [`jsdoc`](https://jsdoc.app) markup language (see Appendix A.1) to document the parameters and the return value of each function.
Because the choosen language for developing the PKM server is Javascript, which is a dynamically typed language, most of the source code has been annotated using the [`jsdoc`](https://jsdoc.app) markup language (see Appendix A.1) to document the parameters and the return value of each function.
The following subsections presents the MongoDB database and the PKM Application Programming Interfaces.
The Git subsystem, which has received a lot of attention, as it is essential software for developers, has Chapter 4 dedicated to it.
......@@ -48,15 +48,15 @@ Figure 4 below outlines the MongoDB collections in each PKM project:<a id="figur
![Figure 4: PKM MongoDB collections](pkm-collections.png)
The `Project` collection hosts the project metadata (name, members).
Each project have a collection for Git working trees metadata, which the PKM server updates after running Git commands.
Each project has a collection for Git working trees metadata, which the PKM server updates after running Git commands.
Some collections track the tool executions (invocations and logs).
The tool specifications enable GUI front-ends to create well-formed tool invocations (for the Process Engine).
The methodology have a collection for the status of each phase of the methodology.
The programming artefacts have both collections for files (both executable binaries and source codes), for compile commands (with compilation flags and options), and (after parsing) for source code Abstract Syntax Trees (ASTs), comments, and annotations.
The models have also both collections for the UML files, and (after parsing) for UML class diagrams and state machines.
The models have also collections for both the UML files, and (after parsing) for UML class diagrams and state machines.
The documentation have a collection for the documentation files (e.g. .docx files), for the Abstract Semi-Formal Models (ASFMs), which Doc to ASFM can generate from .docx files, and for the Graphical documentation written in the Graphical Specification Language (GSL).
Links discovering tool (Semantic parsing) populates the 2D traceability matrix, while Semantic Role Labeling (SRL) and Named Entity Recognition (NER) tools extract information and synthesize Annotations in a dedicated collection.
The TESTAR tool (automated GUI testing) have specialized collections for settings, models, and results.
The TESTAR tool (automated GUI testing) has specialized collections for settings, models, and results.
Finally, there are collections for Common Vulnerabilities and Exposures, and the reviews.
Note that from the PKM REST API point of view the tool specifications, the TESTAR settings, and the methodology status are also properties of the project, even if in reality they are stored in separate collections.
The PKM populates, at project creation, the initial methodology status and a predefined set of tool specifications.
......@@ -240,14 +240,14 @@ The schema of a PKM user from the PKM REST API point of view is the following:
The PKM organizes files per collection depending on the file type. The collections for the PKM files in MongoDB are:
* `'RawSourcecode'` for the source codes
* `'RawSourcecode'` for the source code
* `'RawUML'` for the UML models
* `'RawDocumentation'` for the documentation (e.g. Microsoft Office .docx files)
* `'RawBinaries'` for the executable binaries (e.g. Linux ELF files)
From the PKM REST API point of view, these file collections are shown either as four independent endpoints, or as a one unified endpoint for any file.
For the later endpoint, the PKM guesses the actual underlying collection by applying some rules about the filename extension and the MIME type.
The PKM configuration file `pkm_config.json` allows extending these rules throught the `file_types` property, see Section 2.1.
The PKM configuration file `pkm_config.json` allows extending these rules through the `file_types` property, see Section 2.1.
The schema below for the PKM files in MongoDB can be downloaded from the [pkm-api gitlab repository](https://gitlab.ow2.org/decoder/pkm-api/-/blob/master/api/pkm-db-file-schema.json):
......@@ -313,7 +313,7 @@ The collections for the Abstract Syntax Trees are:
* `'sourcecodeCPP'` for C++
The `sourcecodeC` collection schema is derived from the Frama-C CIL AST.
Due to its extension, here we only show the main structure.
Due to its size, here we only show the main structure.
The complete schema can be downloaded from the [pkm-api gitlab repository](https://gitlab.ow2.org/decoder/pkm-api/-/blob/master/api/pkm-c-source-code-schema.json). The main structure is the following:
{
......@@ -395,7 +395,7 @@ The complete schema can be downloaded from the [pkm-api gitlab repository](https
}
The `sourcecodeCPP` collection schema is derived from the Clang AST.
Due to its extension, here we only show the main structure.
Again, due to its size, here we only show the main structure.
The complete schema can be downloaded from the [pkm-api gitlab repository](https://gitlab.ow2.org/decoder/pkm-api/-/blob/master/api/pkm-cpp-source-code-schema.json). The main structure is the following:
{
......@@ -430,7 +430,7 @@ The complete schema can be downloaded from the [pkm-api gitlab repository](https
}
The `sourcecodeJava` collection schema is derived from the AST of the Eclipse compiler.
Due to its extension, here we only show the main structure.
Again, due to its size, here we only show the main structure.
The complete schema can be downloaded from the [pkm-api gitlab repository](https://gitlab.ow2.org/decoder/pkm-api/-/blob/master/api/pkm-java-source-code-schema.json).
The main structure is the following:
......@@ -1841,7 +1841,7 @@ The main structure of the schema is the following:
}
}
A methodology has several phases, which status consists in the status of each phases of the methodology.
A methodology has several phases, and its global status consists in the status of each phase of the methodology.
The complete schema for the phase status of the methodology can be downloaded from the [pkm-api gitlab repository](https://gitlab.ow2.org/decoder/pkm-api/-/blob/master/api/pkm-methodology-status-schema.json).
The main structure of the schema is the following:
......@@ -1918,9 +1918,9 @@ In *PKM util*:
The source code of each part of the Javascript SDK is on the pkm-api gitlab repository respectively in directory [`core`](https://gitlab.ow2.org/decoder/pkm-api/-/tree/master/core) and [`util`](https://gitlab.ow2.org/decoder/pkm-api/-/tree/master/util).
Most of functions and methods use an asynchronous control flow pattern using javascript Promises.
Concretely that means that every API functions and methods are non-blocking and returns immediately an instance of Class `Promise`, before actual work is finished.
The javascript runtime triggers callbacks once the promise resolves.
Most of functions and methods use an asynchronous control flow pattern using Javascript Promises.
Concretely that means that all API functions and methods are non-blocking and returns immediately an instance of Class `Promise`, before actual work is finished.
The Javascript runtime triggers callbacks once the promise resolves.
Code below shows the caveat when coding with promises:
var expected_result;
......@@ -1939,7 +1939,7 @@ Code below shows the caveat when coding with promises:
// because not yet assigned
...
Developers not used to using javascript promises who would like to extend the PKM server can find more information about javascript promises by consulting the resources below:
Developers not used to using Javascript promises who would like to extend the PKM server can find more information about Javascript promises by consulting the resources below:
* [MDN Web Docs: Promise](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise)
* [Javascript Tutorial: The Definitive Guide to the JavaScript Promises](https://www.javascripttutorial.net/es6/javascript-promises)
......@@ -2057,13 +2057,13 @@ Class `PKM` also provides developer with `insert_documents_spanned`, `update_doc
* `update_documents_spanned(dbName, collection_names, documents, dispatch, signature)`
* `insert_update_documents_spanned(dbName, collection_names, documents, update, dispatch, signature)`
There are also some specialized methods but still quite generic methods named `insert_update_files`, `insert_update_files_spanned` and `delete_files_spanned` to manage files:
There are also some specialized but still quite generic methods named `insert_update_files`, `insert_update_files_spanned` and `delete_files_spanned` to manage files:
* `insert_update_files(dbName, collection_name, files, update, options = {})`
* `insert_update_files_spanned(dbName, collection_names, files, update, dispatch, options = {})`
* `delete_files_spanned(dbName, collection_names, query = {})`
Class `PKM` provides developer with many specialized methods that rely on these generic methods to manage all type of documents and files that PKM supports.
Class `PKM` provides developer with many specialized methods that rely on these generic methods to manage all types of documents and files that PKM supports.
#### 3.3.1.3 Document querying <a id="3.1.3"></a>
......
......@@ -2,7 +2,7 @@
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="date" content='12/10/21'>
<meta name="date" content='12/14/21'>
<title>D1.5 Open source PKM server-side software</title>
<link rel="stylesheet" href='../style/style.css' type="text/css">
</head>
......@@ -37,7 +37,7 @@
<p><strong>Issue 2</strong>. The second issue is that some Git commands can have system-wide paths passed as command line options or parameters. Concretely, we had to find a solution to avoid path out-of-bound situations, for instance with the <code>'clone'</code> command. In fact, for most commands, Git already checks that the paths actually target files in the Git working tree where the command runs. For other commands like <code>'clone'</code>, we have made the PKM parse such commands and then check the value of each options and arguments that Git interprets as a path before spawning Git. Currently only the following commands are parsed: <code>'clone'</code>, <code>'fetch'</code>, <code>'pull'</code>, <code>'push'</code>, <code>'config'</code> and <code>'worktree'</code>. Additional commands of which we are not aware may require special attention.</p>
<p><strong>Issue 3</strong>. The third issue is that some commands requires authentication to a remote server, like <code>'clone'</code>. Indeed, we have had to ensure that a request to the PKM server is not blocked (resulting in no response from the PKM server) because the Git program is waiting credentials forever. This happens because the authentication of Git (one prompt for the user name and one prompt for the password) is interactive. Redirecting the input of Git to provide it the user’s credentials is tricky because Git reads the credentials from the terminal rather a pipe. The solution that we have found is to make the PKM use the <a href="http://sshpass.sourceforge.net"><code>sshpass</code></a> command to provide Git (or ssh) with the credentials. Indeed, when the PKM detects that a command may need the user’s credential, Git runs under the supervision of sshpass that runs itself under the supervision of the PKM server. Because <code>sshpass</code> cannot provide more than one credential for each Git command, the PKM checks the command options to ensure that the command only targets one remote server. When the remote server is just named, i.e. no URL is provided on the command line (e.g. <code>'origin'</code>), the PKM also looks at each <code>remote</code> in the Git local configuration (aka. <code>.git/config</code>). The PKM then checks if it has credentials available (either in the request body or in the user’s “wallet”) for the remote server before actually spawning Git. For HTTP/HTTPS, the PKM provides the user’s name through the URL by rewriting the URL on the fly. As Git may record the given HTTP/HTTPS URLs in the Git config remotes, the PKM make the HTTP/HTTPS URLs anonymous again in the Git config remotes.</p>
<h2 id="users-credential-wallet">4.4 User’s credential “wallet” <a id="4"></a></h2>
<p>Each user has a kind of “wallet” which stores the user’s credential related to Git, either completely or partially (e.g. without the password).. This “wallet” is in property <code>git_user_credentials</code> of PKM user (see Section 3.2.1),</p>
<p>Each user has a kind of “wallet” which stores the user’s credential related to Git, either completely or partially (e.g. without the password). This “wallet” is in property <code>git_user_credentials</code> of PKM user (see Section 3.2.1).</p>
<pre><code>PkmUser
{
&quot;name&quot;: &quot;name&quot;,
......@@ -60,7 +60,7 @@
}
]
}</code></pre>
<p>Only user can view and modify own credentials. It is plain text in the REST API, thus it deserves securing the REST API with HTTPS, which PKM server supports too. It is AES256 encrypted in MongoDB database. The decryption key (32 bytes) is stored permanently in a file named <code>'secret'</code> (configurable in File <code>pkm_config.json</code>) on the server disk (read-only for the PKM system user and forbidden for the group and others) or on same docker volume like the one for File <code>pkm_config.json</code>.</p>
<p>Only user can view and modify their own credentials. It is plain text in the REST API, thus it deserves securing the REST API with HTTPS, which PKM server supports too. It is AES256 encrypted in MongoDB database. The decryption key (32 bytes) is stored permanently in a file named <code>'secret'</code> (configurable in File <code>pkm_config.json</code>) on the server disk (read-only for the PKM system user and forbidden for the group and others) or on same docker volume like the one for File <code>pkm_config.json</code>.</p>
<h2 id="operations">4.5 Operations <a id="5"></a></h2>
<p>The PKM REST API has support for the following operations related to the Git support:</p>
<p><strong>Run a Git command sequence</strong></p>
......@@ -136,7 +136,7 @@ DELETE /git/working_trees/{dbName}/{gitWorkingTree}?dontDeletePkmFiles=…</code
<p>These operations allow deleting Git working trees created with <code>'clone'</code>. These operations can also delete the corresponding files in the PKM. There is no other way to delete Git working trees, and the only way to create a new Git working tree is to run a <code>clone</code> command.</p>
</div>
<div id="footer">
<span><a href="../index.html">D1.5 Open source PKM server-side software</a> - 12/10/21</span>
<span><a href="../index.html">D1.5 Open source PKM server-side software</a> - 12/14/21</span>
</div>
</body>
</html>
......@@ -65,8 +65,8 @@ As Git may record the given HTTP/HTTPS URLs in the Git config remotes, the PKM m
## 4.4 User's credential "wallet" <a id="4"></a>
Each user has a kind of "wallet" which stores the user's credential related to Git, either completely or partially (e.g. without the password)..
This "wallet" is in property `git_user_credentials` of PKM user (see Section 3.2.1),
Each user has a kind of "wallet" which stores the user's credential related to Git, either completely or partially (e.g. without the password).
This "wallet" is in property `git_user_credentials` of PKM user (see Section 3.2.1).
PkmUser
{
......@@ -91,7 +91,7 @@ This "wallet" is in property `git_user_credentials` of PKM user (see Section 3.2
]
}
Only user can view and modify own credentials.
Only user can view and modify their own credentials.
It is plain text in the REST API, thus it deserves securing the REST API with HTTPS, which PKM server supports too.
It is AES256 encrypted in MongoDB database.
The decryption key (32 bytes) is stored permanently in a file named `'secret'` (configurable in File `pkm_config.json`) on the server disk (read-only for the PKM system user and forbidden for the group and others) or on same docker volume like the one for File `pkm_config.json`.
......
......@@ -2,7 +2,7 @@
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="date" content='12/10/21'>
<meta name="date" content='12/14/21'>
<title>D1.5 Open source PKM server-side software</title>
<link rel="stylesheet" href='../style/style.css' type="text/css">
</head>
......@@ -86,7 +86,7 @@
<p>Appendix A.2 contains detailed explanations about implementation design of the REST server that provides developer with the REST API.</p>
</div>
<div id="footer">
<span><a href="../index.html">D1.5 Open source PKM server-side software</a> - 12/10/21</span>
<span><a href="../index.html">D1.5 Open source PKM server-side software</a> - 12/14/21</span>
</div>
</body>
</html>
......@@ -2,7 +2,7 @@
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="date" content='12/10/21'>
<meta name="date" content='12/14/21'>
<title>D1.5 Open source PKM server-side software</title>
<link rel="stylesheet" href='../style/style.css' type="text/css">
</head>
......@@ -18,11 +18,11 @@
<div id="content">
<h1 id="conclusion">6 Conclusion</h1>
<p>This document has described how to install the PKM. It has also described the implementation of the PKM server that is the software architecture, the built-in support for Git, and the parsers.</p>
<p>There are some lessons that the writer has learned while developing the PKM server. The MongoDB database server, despite some technical limitations, is easy to use, and its speed is quite good even with a huge amount of complex programming artefacts. Node.js is a very good piece of software, with a huge choice of packages, which execution speed is also very good considering that javascript is a dynamically typed language. The OpenAPI specification standard is very convenient to document RESTful APIs when everything is JSON and to generate automatically the SDK for the clients. However, in our experiments, due to the supporting tools (Swagger codegen and OpenAPI generator) limitations and bugs, it has proved to be inconvenient for anything else than JSON (e.g. direct file upload/download, plain text), or with complex JSON schemas (some of them are disabled in the PKM REST API, but still enforced before reaching the database).</p>
<p>There are some lessons that the writer has learned while developing the PKM server. The MongoDB database server, despite some technical limitations, is easy to use, and its speed is quite good even with a huge amount of complex programming artefacts. Node.js is a very good piece of software, with a huge choice of packages, which execution speed is also very good considering that Javascript is an interpreted language. The OpenAPI specification standard is very convenient to document RESTful APIs when everything is JSON and to generate automatically the SDK for the clients. However, in our experiments, due to the supporting tools (Swagger codegen and OpenAPI generator) limitations and bugs, it has proved to be inconvenient for anything else than JSON (e.g. direct file upload/download, plain text), or with complex JSON schemas (some of them are disabled in the PKM REST API, but still enforced before reaching the database).</p>
<p>The PKM server have also its own limitations that deserve attention in future developments. The PKM files, and more generally the artefacts in the PKM, have no revision numbers so that it is not easy to track progressive enhancement of the knowledge in the PKM. The PKM users management, especially concerning the user’s roles, is tightly geared to the MongoDB user’s management, which is database oriented, but it is not necessarily convenient for a SaaS approach. Direct upload (as attachment in a POST HTTP request) toward the database is unsupported due to the limitations of OpenAPI supporting tools.</p>
</div>
<div id="footer">
<span><a href="../index.html">D1.5 Open source PKM server-side software</a> - 12/10/21</span>
<span><a href="../index.html">D1.5 Open source PKM server-side software</a> - 12/14/21</span>
</div>
</body>
</html>
......@@ -5,7 +5,7 @@ It has also described the implementation of the PKM server that is the software
There are some lessons that the writer has learned while developing the PKM server.
The MongoDB database server, despite some technical limitations, is easy to use, and its speed is quite good even with a huge amount of complex programming artefacts.
Node.js is a very good piece of software, with a huge choice of packages, which execution speed is also very good considering that javascript is a dynamically typed language.
Node.js is a very good piece of software, with a huge choice of packages, which execution speed is also very good considering that Javascript is an interpreted language.
The OpenAPI specification standard is very convenient to document RESTful APIs when everything is JSON and to generate automatically the SDK for the clients.
However, in our experiments, due to the supporting tools (Swagger codegen and OpenAPI generator) limitations and bugs, it has proved to be inconvenient for anything else than JSON (e.g. direct file upload/download, plain text), or with complex JSON schemas (some of them are disabled in the PKM REST API, but still enforced before reaching the database).
......
......@@ -2,7 +2,7 @@
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="date" content='12/10/21'>
<meta name="date" content='12/14/21'>
<title>D1.5 Open source PKM server-side software</title>
<link rel="stylesheet" href='../style/style.css' type="text/css">
</head>
......@@ -20,11 +20,11 @@
<h2 id="a.1-javascript-sdk-documentation">A.1 Javascript SDK documentation <a id="1"></a></h2>
<p>The Javascript SDK documentation (generated with jsdoc) is available at: <a href="https://decoder.ow2.io/pkm-api/public/sdk/index.html">https://decoder.ow2.io/pkm-api/public/sdk/index.html</a></p>
<h2 id="a.2-openapi-generator">A.2 OpenAPI generator <a id="2"></a></h2>
<p><a href="https://github.com/OpenAPITools/openapi-generator">OpenAPI Generator</a> allows generation of API client libraries (SDK generation), server stubs, documentation and configuration automatically given an OpenAPI Specification. Directory <a href="https://gitlab.ow2.org/decoder/pkm-api/-/blob/master/openapi_gen">openapi_gen</a> contains some generation scripts which leverages on this tool to fully generate server (not only server stubs), SDK for the clients written in javascript and REST API documentation. These scripts take a single argument which is a configuration file:</p>
<p><a href="https://github.com/OpenAPITools/openapi-generator">OpenAPI Generator</a> allows generation of API client libraries (SDK generation), server stubs, documentation and configuration automatically given an OpenAPI Specification. Directory <a href="https://gitlab.ow2.org/decoder/pkm-api/-/blob/master/openapi_gen">openapi_gen</a> contains some generation scripts which leverages on this tool to fully generate server (not only server stubs), SDK for the clients written in Javascript and REST API documentation. These scripts take a single argument which is a configuration file:</p>
<ul>
<li><code>generate_server.sh</code>: generate the REST server and the documentation of the SDKs for the clients</li>
<li><code>generate_client.sh</code>: generate the SDK for a javascript client</li>
<li><code>generate.sh</code>: generate both the REST server and the SDK for a javascript client</li>
<li><code>generate_client.sh</code>: generate the SDK for a Javascript client</li>
<li><code>generate.sh</code>: generate both the REST server and the SDK for a Javascript client</li>
</ul>
<p>The configuration file contains configuration parameter value assignments (<code>PARAM=value</code>).</p>
<p>The configuration parameters are the following:</p>
......@@ -35,12 +35,12 @@
<li><code>SERVER_COMPRESSION_LEVEL</code>: gzip compression level of server responses (1=fast to 9=best, default: 5)</li>
<li><code>SERVER_API_SPEC</code>: OpenAPI specification file of the server (defaut: <code>'api/openapi.yaml'</code>)</li>
<li><code>SERVER_API_SCHEMAS</code>: External JSON schema files referenced by the OpenAPI specification file – e.g. <code>(schema1.json schema2.json schema3.json)</code>, default: <code>()</code></li>
<li><code>SERVER_REQUIRES</code>: Javascript code snippet intended for javascript requires statements (e.g. <code>const PKM = require('pkm')</code>, default: none)</li>
<li><code>SERVER_REQUIRES</code>: Javascript code snippet intended for Javascript requires statements (e.g. <code>const PKM = require('pkm')</code>, default: none)</li>
<li><code>SERVER_CONFIG</code>: Run-time configuration file of the server (e.g. <code>'server_config.json'</code>, default: none)</li>
<li><code>SERVER_DIR</code>: relative path to the directory where to generate the full server source code (default: <code>'server'</code>)</li>
<li><code>CLIENT_SDK_DIR</code>: relative path to the directory where to generate the SDK for the client written in javascript (default: <code>'client'</code>)</li>
<li><code>CLIENT_SDK_DIR</code>: relative path to the directory where to generate the SDK for the client written in Javascript (default: <code>'client'</code>)</li>
</ul>
<p>The paths are all relative to directory containing the configuration file. The actual implementation of each API functions shall be in sub-directory <code>services</code>.</p>
<p>The paths are all relative to directory containing the configuration file. The actual implementation of each API function shall be in sub-directory <code>services</code>.</p>
<p>Figure 7 below shows the OpenAPI generation flow:<a id="figure7"></a></p>
<figure>
<img src="./openapi_gen.png" alt="Figure 7: OpenAPI generation flow" /><figcaption aria-hidden="true">Figure 7: OpenAPI generation flow</figcaption>
......@@ -55,7 +55,7 @@
<li>UNISIM Excavator for PKM</li>
</ul>
<h2 id="a.3-integration-non-regression-testing">A.3 Integration &amp; non-regression testing <a id="3"></a></h2>
<p>There is a set of benchmarks for the DECODER project. These benchmarks are tests mostly intended for integration testing and non-regression testing of the DECODER Project tool-chain. There are not only basic tests but also more complex tests with software from the real world. They have been used extensively to ensure that the server is performing its missions correctly and scaling up in realistic scenarios.</p>
<p>There is a set of benchmarks for the DECODER project. These benchmarks are tests mostly intended for integration testing and non-regression testing of the DECODER Project tool-chain. There are not only basic tests but also more complex tests with software from the real world, such as OpenCV and Linux. They have been used extensively to ensure that the server is performing its missions correctly and scaling up in realistic scenarios.</p>
<p>These benchmarks are available at <a href="https://gitlab.ow2.org/decoder/integration-tests">https://gitlab.ow2.org/decoder/integration-tests</a>.</p>
<p>Each test has a bash script (<code>run.sh</code>) to run the test and a dataset (in directory <code>data</code>).</p>
<p>The currently available tests are:</p>
......@@ -73,7 +73,7 @@
<p>The tests use the popular <a href="https://curl.se/">cURL</a> program to communicate with the PKM and tools through their REST APIs.</p>
</div>
<div id="footer">
<span><a href="../index.html">D1.5 Open source PKM server-side software</a> - 12/10/21</span>
<span><a href="../index.html">D1.5 Open source PKM server-side software</a> - 12/14/21</span>
</div>
</body>
</html>
......@@ -7,12 +7,12 @@ The Javascript SDK documentation (generated with jsdoc) is available at: [https:
## A.2 OpenAPI generator <a id="2"></a>
[OpenAPI Generator](https://github.com/OpenAPITools/openapi-generator) allows generation of API client libraries (SDK generation), server stubs, documentation and configuration automatically given an OpenAPI Specification.
Directory [openapi_gen](https://gitlab.ow2.org/decoder/pkm-api/-/blob/master/openapi_gen) contains some generation scripts which leverages on this tool to fully generate server (not only server stubs), SDK for the clients written in javascript and REST API documentation.
Directory [openapi_gen](https://gitlab.ow2.org/decoder/pkm-api/-/blob/master/openapi_gen) contains some generation scripts which leverages on this tool to fully generate server (not only server stubs), SDK for the clients written in Javascript and REST API documentation.
These scripts take a single argument which is a configuration file: