4.5. Data Locations

To import data from reports in MQC, the data can be provided as files. These files can be located in the local file system, a network path or a git repository.

Alternativly files or data can be fetched from an API. (see Application Programming Interface)

Figure 4.37 Data locations in MQC. By clicking on “Add” a new data location can be configured.

A new data location can be added through a browser in the data location dialog. The type of the data location must be selected first and then the respective directory, files or git repository can be added. You can add multiple data locations to your project.

Figure 4.38 The Browser to add a new Data Location

If the monitoring is enabled in the settings, the data locations are constantly watched for updates and changes (see Monitor Data Locations for changes).

In this case the user is notified about the changes in the toolbar or the project is automatically updated.

The data locations can also be refreshed independent of the monitoring, by clicking on the Refresh Button in the footer of the Data Locations Dialog.

Figure 4.39 Refresh button in the Data Location Dialog

4.5.1. Local File System / Network / Sample Projects

The local file system and network types function similarly. A directory, and therefore all subdirectories and files within, or a single file can be selected as the data location. At any point, the allowed file extensions that can be read by MQC are displayed below.

Navigating through the file system works like a windows explorer. Double-clicking a directory opens it. The current path is displayed at the top of the browser and can be used to navigate upwards in the file system. You can also add the path manually.

Figure 4.40 Local File System selected. Add/Edit the path manually.

Figure 4.41 Network selected. The path can be used to navigation upwards by click.

The local file system of the server is not accessible when using the web player.

You have also direct access to the MQC Showcase source files under Sample Projects (see Figure 4.38).

4.5.2. Version Control Systems

MQC allows version control systems to be used as a data location. Currently only Git is supported.

4.5.2.1. Git

After selecting Git as data location type, a Git repository URL can be added. In case you have multiple repositories with a similar structure, you can configure them as one data location. Use the Add Repository button to extend the list of repository URLs or paste a list of repositories to the edit field, which automatically leads to multiple lines (see Figure 4.42).

Each repository is checked then for availability and accessibility separately. If an authentication is required, you will be asked for your credentials.

If git is not installed, locally (if using the MQC desktop client) or on the server (if using the web player), an error is shown.

Git for windows can be found here: https://git-scm.com/downloads

Git repositories can be added by ssh or https urls.

An rsa private key can be provided for authentication of ssh urls if it is not configured as the default ssh key for the current user.

For an https url the authentication must be done by entering the username and password, if it is not already stored in the windows credential store.

Figure 4.42 Git Repositories dialog to add, edit and remove git repository(ies) to be configured as one data location.

If the configured repository(ies) contain multiple branches, you can select a branch via the drop down next to the list of repository URLs.

Only branches that exist in all repositories can be selected, when multiple Git repositories are given.

Since your commit history might contain commits not relevant for your MQC quality analysis, you have the possibility of filtering these commits in the Commit Filters section of this dialog (see Figure 4.43).

Figure 4.43 Configuration options to add a commit filter.

The Commit Filters section provides three possible targets to filter, namely, Message, Author and Tag. You can further choose to either include or exclude the commits matching this filter by clicking on the toggle button below Apply. The green colored + stands for include and the red colored - stands for exclude. Define a suitable regular expression in the RegEx text box. All the filters to the same target are connected with an OR whereas different targets are connected with an AND.

Figure 4.44 shows a filter that matches all commits From the Author "Jenkins Slave" AND with (Message "Static Analysis" OR Message "Dynamic Analysis").

Figure 4.44 Commit Filters with “AND” between different targets and “OR” between the same targets.

Additionally, MQC offers the possibility to filter by time (see Figure 4.43). This allows to consider only commits of a specific time frame. First and last commit time can be configured separately by:

• using the start and end date of the project (as configured within the Project configuration, which is also the default if nothing is set at all)

• selecting a specific date via the date picker.

If no project start and end date has been configured, all commits are fetched from the repository(ies).

If a commit appears outside the configured project time frame, you will be notified about such commits within the Notifications panel. If such a commit is relevant for the project, the project start and end configuration has to be adapted. Afterwards, a data refresh is necessary to fetch the additional commits from the repository.

With the checkbox shown in Figure 4.45 you can define which date and time is used as the report date-time:

• Git commit date (checked)

• Report creation date time as contained in the report itself (unchecked)

Figure 4.45 Enable checkbox to use commit time as report date.

In addition to the commit filters, it is also possible to select specific folders or files relevant to your analysis. MQC provides two possibilities to do this.

Figure 4.46 Selection menu to choose the folder or file path of relevant data by file system or add patterns to match relevant folder or file paths.

If you select the Regular Expressions option in the drop down, then you can add multiple regular expressions similar to the Commit Filters section.

Figure 4.47 Regular Expression pattern matching for selecting data location paths.

If you select Browser Selection option, one file or folder can be chosen as the data location, similar to selecting directories from a local or a network drive as data location.

Figure 4.48 Browser Selection for choosing data location path.

The Browser Selection is only available when using a single Git repository as data location. For multiple repositories only Regular Expressions could be applied. Additionally, to be able to navigate within and to select specific folders, it is necessary to fetch the repository first by using the Fetch repository button in the left-right corner (see Figure 4.48).

Since the version 6.3, MQC supports the partial checkout feature (sparse checkout) provided by Git. Using sparse checkout, no full clone of the git repository is performed. Only the relevant files and directories based on the defined filters are downloaded from the server.

Using the sparse checkout must be enabled by an administrator. The corresponding configuration is described in the Server Administration Guide.

If sparse checkout is enabled, no browser selection is possible. Additionally, it is not possible to define “exclude” filters for directories as shown in Figure 4.47. Instead of regular expressions for directory paths, the names of the directories have to be used as filter.

If a git data location has been added, a monitoring service periodically checks if a new commit is made on the remote. The default for the check interval is set to 10 seconds. In case of multiple repositories, the supervision timer is stepwise increased for performance reasons based on the number of repositories. I.e. If you have configured more than 100 repositories, the check happens only every 60 seconds.

When data is refreshed, the git repository is updated and all remote changes are fetched and applied to the MQC project. In case some hidden changes were made in your Git repository, use the Force Refresh button (see Figure 4.37) to not just apply changes to a project but to re-import all data anew.

4.5.3. Application Programming Interface

MQC allows Application Programming Interfaces, short APIs, to be used as data locations.

API Connectors will download files, to be read by File Readers, or create data directly.

4.5.3.1. GitLab API

You can select from two different GitLab API connectors in the Connector dropdown of the API-tab of the Add Data Location dialog. For more information see GitLab in API Connectors.

The configuration of these api connectors is identical in many aspects.

Figure 4.49 Add GitLab Pipeline data location in MQC.

• Url

  Url to GitLab. Either self-managed GitLab root or “https://gitlab.com”

• SSL / TLS Authentication

  If the remote server requires authentication with a X.509 client certificate, the certificate file and optional password can be provided.

• Access Token

  Access tokens can be created at three different levels.

  • Personal Access Token: “{URL}/-/profile/personal_access_tokens”

  • Group Access Token: “{URL}/groups/{GROUP}/-/settings/access_tokens”

  • Project Access Token: “{URL}/{GROUP}/{PROJECT}/-/settings/access_tokens”

  When creating an access token, the “read_api” permission must be checked. Once created, the token is only visible once, so be sure to write it down.

  

  Figure 4.50 Create a GitLab Project Access Token.

• Project Branches

  At least one project and it’s corresponding branch must be selected. If you have multiple repositories with similar repository structures or pipelines, you can add more project-branch combinations.

  

  Figure 4.51 Multiple Gitlab Project Branches.

  • Group

    In GitLab, groups are used to manage one or more related projects. Selecting a group via the group dropdown helps to reduce the number of options for project selection, especially if the number of relevant Gitlab projects is very large.

  • Project

    The project dropdown allows you to select from any project that the provided access token has read permissions on the gitlab server.

  • Branch

    The branch dropdown lets you choose from available branches in the selected project.

• Date Filter

  Pipelines can be filtered by date range.

  • From

    Choose to use the project start date (see Project configuration) as the From date, or enter a specific date.

  • To

    Choose to use the project end date (see Project configuration) as the To date, or enter a specific date.

4.5.3.1.1. Pipeline

Configuration of the data location specific to the Pipeline Connectors.

• Pipeline Types

  Multiple types of pipelines can run in a project.

  

  Figure 4.52 GitLab Pipeline Types

  • Branch Pipeline:

    Pipeline that runs every time changes are pushed to the Branch.

    (see Branch Pipeline in GitLab Docs)

  • Merge Request Pipeline:

    Pipeline that runs every time changes are pushed to the Source Branch of a Merge Request.

    Only the last Pipeline of a Merge Request, that was merged into the selected Project Branch, is imported.

    (see Merge Request Pipelines in GitLab Docs)

• Load Types

  • Artifacts:

    Download the artifact files and import them with the file readers.

  • Test Reports:

    Load the JUnit and Coverage result information from the Gitlab test reports directly. In case “Test Reports” is selected as load type, a name for the artifact must be defined for each project branch. This name is only used by MQC to assign the data and findings to the artifact.

  

  Figure 4.53 Project Branch with Artifact Name for GitLab Pipeline Test Report

• Job Filters

  Job filters can be used to restrict the jobs from which pipeline artifacts or test reports are retrieved.

  All include filters of the same type are combined with OR, while include filters of different types are combined with AND. Exclude filters always take precedence.

  You can preview the most recent jobs that match these filters in the vertical dialog box next to the Add Data Location dialog box.

  

  Figure 4.54 GitLab Job Filter on the Stage “test”

• Report Date

  Which time should be used when extracting the data.

  • Use report time

    The date and time from the report file will be used when read by the file reader.

  • Use pipeline creation time

    The date and time the pipeline was created is used. (e.g. triggered by a push)

  • Use job finished time

    The date and time the job was completed is used. (after the report files have been generated)

4.5.3.1.2. Repository

• Commit Filters

  Commit filters can be used to restrict the commits from which files are retrieved.

  All include filters of the same type are combined with OR, while include filters of different types are combined with AND. Exclude filters always take precedence.

  You can preview the most recent commits that match these filters in the vertical dialog box next to the Add Data Location dialog box.

  

  Figure 4.55 GitLab Commit Filter with the Message as “Gitlab Pipeline”

• File Filters

  File filters can be used to restrict the files retrieved.

  All include filters of the same type are combined with OR, while include filters of different types are combined with AND. Exclude filters always take precedence.

  Only directories can be used to filter files. Partial directory names or wildcards are not allowed. (e.g. “MyDirectoryName”, “MyDirectoryName/MySubDirectoryName”)

  You can preview the files that match these filters in the vertical dialog box next to the Add Data Location dialog box.

  

  Figure 4.56 GitLab File Filters for the directories 30_StaticAnalysis and 40_DynamicAnalysis

4.5.3.2. Jenkins API

When selecting the Jenkins API connector in the Connector dropdown of the API-tab of the Add Data Location dialog, two different ways to load data are supported. For more information see Jenkins in API Connectors.

For both the configuration of the api connector is identical in many aspects.

Figure 4.57 Add Jenkins Pipeline data location in MQC.

• Url

  Url to Jenkins.

• SSL / TLS Authentication

  If the remote server requires authentication with a X.509 client certificate, the certificate file and optional password can be provided.

• Username

  The username of the Account that should be used for the API access. We recommend a specific account for API requests.

• API Token

  An API Token can be created in the user account (see username).

  

  Figure 4.58 API Token in a Jenkins user account

• Jobs

  At least one job must be selected.

  

  Figure 4.59 Jenkins Jobs.

  • Job Name

    The job dropdown allows you to select from any job that the provided API token has read permissions on the jenkins server.

• Load Types

  • Artifacts:

    Download the artifact files and import them with the file readers.

  • Test Reports:

    Load the JUnit result information from the Jenkins test reports directly. In case “Test Reports” is selected as load type, a name for the artifact must be defined for each Jenkins Job. This name is only used by MQC to assign the data and findings to the artifact.

  

  Figure 4.60 Job with Artifact Name for Jenkins Pipeline Test Report

• Date Filter

  Pipelines can be filtered by date range.

  • From

    Choose to use the project start date (see Project configuration) as the From date, or enter a specific date.

  • To

    Choose to use the project end date (see Project configuration) as the To date, or enter a specific date.

• Report Date

  Which time should be used when extracting the data.

  • Use build time

    If selected, date and time of the build is used, else the date and time from the report file will be used when read by the file reader.

4.5.3.3. Jira API

You can select Jira Issues from the Connector dropdown in the API tab of the Add Data Location dialog to read data from a Jira project. For more information, see Jira in API Connectors.

The configuration process starts with the Connection section.

Figure 4.61 Add Jira Issues data location in MQC.

The Connection configuration includes the following details:

• Url

  The base URL of your Jira instance, typically in the format https://yourcompany.atlassian.net.

• SSL / TLS Authentication

  If the remote server requires authentication with a X.509 client certificate, the certificate file and optional password can be provided.

• Authentication

  Select Basic as the authentication type. Basic authentication requires you to encode your username and API token. Jira uses this combination for secure access.

• Username

  Enter your Jira username, which is typically the email address you use to log in to Jira.

• API Token

  You can create an API token from Jira by navigating to: Jira > Atlassian account settings > Security.

  

  Figure 4.62 API Token in a Jira account settings

For a single connection to the Jira server, you can define different Import configurations that read data in various ways. To add additional import configurations, click the Add Import button.

Figure 4.63 Add Import configuration to the Jira issues connector

The Import section contains three different subsections:

4.5.3.3.1. Setup

The Setup subsection consists of the following elements:

• Project

  The project dropdown allows you to select from any project that you have access to, based on your connection to the Jira server.

• Issue Type

  The Issue Type dropdown allows you to select from any issue type defined for the selected project.

• Import Granularity

  Here, you define the granularity for reading the change log of the issues with the selected type. The most common use case is to match the project revision granularity by enabling the Use project revision granularity checkbox. Alternatively, you can choose an option from the dropdown.

  

  Figure 4.64 Setup subsection in Jira issues API connector

4.5.3.3.2. Filters

Filters allow you to narrow down issues for a chosen issue type, along with their associated change logs. For filtering issues, the selected fields are checked against the defined constraints. Based on this check, each issue either passes or fails the filter, ensuring that only the relevant issues are displayed.

For each filter, choosing Issue as the target means that the entire issue will be included or excluded if the filter criteria are met at any time throughout the issue’s history. On the other hand, selecting Issue Version as the target will include or exclude only those specific changes within the issue that satisfy the filter conditions.

Include filters of the same type are combined using OR, while include filters of different types are combined using AND. Exclude filters always take priority over include filters.

You can view a preview of the issues that match these filters in the vertical dialog box adjacent to the Add Data Location dialog box.

Figure 4.65 Filters section in Jira issues API connector

4.5.3.3.3. Mappings

In the mapping section, you will see all the items that need to be defined for creating findings from issues.

Figure 4.66 Mappings section in Jira issues API connector

Each item includes a switch that indicates if it is simple text or should be configured to read from one of the fields of the issue. As text, the value should be defined in the textbox.

Figure 4.67 Static text used as Measure value

Alternatively, an item can be configured to read from one of the fields of the issue. To do this, select the appropriate field name from the dropdown. If you need to extract specific data from that field, you must define the regex pattern and the replacement value accordingly. If not, you can leave them unchanged.

Figure 4.68 Static text used as Measure value

For Artifact Inner Path, Subject Path for Artifact, and Subject Path, which can have multiple parts, the configuration allows you to define additional fields for reading the next parts. The optional items can remain empty.

4.5.3.4. JFrog Artifactory API

The JFrog Artifactory API Connector let’s you download files to be read by File Readers.

For more information see JFrog Artifactory in API Connectors.

Figure 4.69 Add JFrog Artifactory Artifacts data location in MQC.

• Url

  Url to JFrog Artifactory.

• SSL / TLS Authentication

  If the remote server requires authentication with a X.509 client certificate, the certificate file and optional password can be provided.

• Identity Token

  An Identity Token can be created in the user profile.

  

  Figure 4.70 JFrog Artifactory User Profile with “Generate an Identity Token” Button

• Repositories

  At least one repository must be selected.

  

  Figure 4.71 Project selection is required before the repository selection.

  • Project

    The project dropdown allows you to select from any project that the provided Identity Token has read permissions on the server. Additionally, “All” can be selected, to not filter by project. If the user has no permissions to view projects, the dropdown is not shown.

    

    Figure 4.72 No access to projects, Project-Dropdown is not available and Repository dropdown is active.

  • Repository

    The repository dropdown allows you to select from any repository that the provided Identity Token has read permissions on the server, filtered by the selected project.

    

    Figure 4.73 After selecting a project, a repository has to be selected.

• File Filters

  File filters can be used to restrict the files retrieved.

  All include filters of the same type are combined with OR, while include filters of different types are combined with AND. Exclude filters always take precedence.

  You can preview the files that match these filters in the vertical dialog box next to the Add Data Location dialog box.

  

  Figure 4.74 JFrog File Filters for the path 30_StaticAnalysis in 2023

• Monitoring

  The fastest way to monitor for changes in JFrog Artifactory is to only check for new or changed files. If you plan to delete files in JFrog, this checkbox should be enabled.

  

  Figure 4.75 Check for deleted files while monitoring