feat(jans-pycloudlib): add support reading configuration from file #9037

iromli · 2024-07-26T21:21:58Z

Prepare

Read PR guidelines
Read license information

Description

Overview:

password files for persistence are optional, they will be pre-populated from local/remote secrets if files are missing
Vault RoleID and SecretID files are optional, they will be pre-populated from local/remote secrets if files are missing
changes to charts will be added in next PRs to avoid huge diff with main branch

Target issue

closes #9035

Implementation Details

Test and Document the changes

Static code analysis has been run locally and issues have been fixed
Relevant unit and integration tests have been added/updated
Relevant documentation has been updated if any (i.e. user guides, installation and configuration guides, technical design docs etc)

Please check the below before submitting your PR. The PR will not be merged if there are no commits that start with docs: to indicate documentation changes or if the below checklist is not selected.

I confirm that there is no impact on the docs due to the code changes in this PR.

Signed-off-by: iromli <isman.firmansyah@gmail.com>

dryrunsecurity · 2024-07-26T21:22:16Z

DryRun Security Summary

This pull request covers a wide range of security and reliability improvements across multiple components of the Jans application suite, including secure persistence configuration, credential and secrets management, configuration updates, data migration, dependency management, and startup/initialization processes.

Expand for full summary

Summary:

The code changes in this pull request cover a wide range of updates and improvements to the Jans application's security and reliability. The changes span multiple components, including the Jans Auth Server, Jans Casa, Jans Config API, Jans FIDO2, Jans Keycloak Link, Jans Link, and the Jans Persistence Loader.

The key security-related aspects of these changes include:

Secure Persistence Configuration: The code ensures that the application's persistence layer (LDAP, SQL, Couchbase, Spanner) is properly configured and that sensitive information, such as passwords and certificates, is securely managed and synchronized across the different backends.
Secure Credential and Secrets Management: The code handles the generation, storage, and synchronization of various credentials and secrets, including client IDs, client secrets, user passwords, and SSL/TLS certificates, to maintain the application's overall security.
Secure Configuration Updates: The code includes mechanisms to update the application's dynamic, static, and error configurations during the upgrade process, ensuring that the configurations remain secure and up-to-date.
Secure Data Migration: The code handles the migration of data, such as SCIM entries, profiles, and client configurations, during the upgrade process, ensuring that the application's security posture is maintained.
Secure Dependency Management: The code manages the application's dependencies, including the extraction and inclusion of common libraries, to ensure that the application is using secure and up-to-date components.
Secure Startup and Initialization: The code includes various checks and validation steps to ensure that the application's startup and initialization processes are secure and reliable, such as waiting for the persistence layer to be available and validating the configuration settings.

Overall, the code changes in this pull request demonstrate a strong focus on improving the security and reliability of the Jans application suite, with a particular emphasis on secure persistence management, credential handling, configuration updates, and data migration processes.

Files Changed:

The changes in this pull request span multiple files across the Jans application suite, including:

docker-jans-auth-server/scripts/bootstrap.py: Handles the configuration and setup of the Jans Auth Server's persistence layer, logging, and SSL/TLS certificates.
docker-jans-auth-server/scripts/upgrade.py: Updates the dynamic configuration and client scopes for the Lock feature in the Jans Auth Server.
docker-jans-auth-server/scripts/wait.py: Removes the wait_for_persistence function call, potentially optimizing the startup process.
docker-jans-auth-server/scripts/mod_context.py: Manages the extraction and inclusion of common libraries for the Jans Auth Server.
docker-jans-casa/scripts/bootstrap.py: Configures the persistence layer, secrets management, and logging for the Jans Casa application.
docker-jans-casa/scripts/mod_context.py: Updates the import path for the PersistenceMapper class.
docker-jans-casa/scripts/wait.py: Removes the wait_for_persistence function call.
docker-jans-casa/scripts/upgrade.py: Updates the dynamic, static, and error configurations for the FIDO2 module in the Jans Casa application.
docker-jans-config-api/scripts/mod_context.py: Manages the extraction and inclusion of common libraries for the Jans Config API application.
docker-jans-config-api/scripts/bootstrap.py: Configures the persistence layer, SSL/TLS certificates, and logging for the Jans Config API application.
docker-jans-config-api/scripts/upgrade.py: Updates the API dynamic configuration, client scopes, and other settings during the upgrade process.
docker-jans-fido2/scripts/bootstrap.py: Configures the persistence layer, SSL/TLS certificates, and logging for the Jans FIDO2 application.
docker-jans-fido2/scripts/mod_context.py: Updates the import path for the PersistenceMapper class.
docker-jans-fido2/scripts/wait.py: Removes the wait_for_persistence_conn function call.
docker-jans-fido2/scripts/upgrade.py: Updates the dynamic, static, and error configurations

Code Analysis

We ran 9 analyzers against 30 files and 1 analyzer had findings. 8 analyzers had no findings.

Analyzer	Findings
Authn/Authz Analyzer	20 findings

Riskiness

🟢 Risk threshold not exceeded.

View PR in the DryRun Dashboard.

jans-pycloudlib/jans/pycloudlib/manager.py

Signed-off-by: iromli <isman.firmansyah@gmail.com>

sonarqubecloud · 2024-08-20T14:39:02Z

Quality Gate passed for 'agama parent'

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

sonarqubecloud · 2024-08-20T14:39:36Z

Quality Gate passed for 'keycloak-integration-parent'

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

sonarqubecloud · 2024-08-20T14:41:03Z

Quality Gate passed for 'orm'

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

Signed-off-by: iromli <isman.firmansyah@gmail.com>

…guration.json is missing Signed-off-by: iromli <isman.firmansyah@gmail.com>

Signed-off-by: iromli <isman.firmansyah@gmail.com>

sonarqubecloud · 2024-08-26T22:36:38Z

Quality Gate passed for 'jans-pycloudlib'

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

Signed-off-by: Isman Firmansyah <iromli@users.noreply.github.com>

sonarqubecloud · 2024-08-27T15:22:09Z

Quality Gate passed for 'jans-cli'

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

sonarqubecloud · 2024-08-27T15:23:37Z

Quality Gate passed for 'Fido2 API'

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

sonarqubecloud · 2024-08-27T15:23:43Z

Quality Gate passed for 'jans-config-api-parent'

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

sonarqubecloud · 2024-08-27T15:24:08Z

Quality Gate passed for 'Jans-Keycloak-Link'

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

sonarqubecloud · 2024-08-27T15:24:15Z

Quality Gate passed for 'jans-core'

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

sonarqubecloud · 2024-08-27T15:24:17Z

Quality Gate passed for 'jans-linux-setup'

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

sonarqubecloud · 2024-08-27T15:25:19Z

Quality Gate passed for 'SCIM API'

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

…9037) * feat(jans-pycloudlib): add support reading configuration from file Signed-off-by: iromli <isman.firmansyah@gmail.com> * tests(jans-pycloudlib): adjust testcases Signed-off-by: iromli <isman.firmansyah@gmail.com> * feat(jans-pycloudlib): add CLI command to generate configuration spec Signed-off-by: iromli <isman.firmansyah@gmail.com> * refactor: simplified configmaps and secrets adapters Signed-off-by: iromli <isman.firmansyah@gmail.com> * fix: handle missing params Signed-off-by: iromli <isman.firmansyah@gmail.com> * fix: add backward-compat for configuration and dump files Signed-off-by: iromli <isman.firmansyah@gmail.com> * test(jans-pycloudlib): fix transform_data testcase Signed-off-by: iromli <isman.firmansyah@gmail.com> * refactor: handle mounted password files Signed-off-by: iromli <isman.firmansyah@gmail.com> * refactor: handle mounted files in OCI images Signed-off-by: iromli <isman.firmansyah@gmail.com> * fix: ensure couchbase password files are exist Signed-off-by: iromli <isman.firmansyah@gmail.com> * refactor: generate opendj.pkcs12 on-the-fly Signed-off-by: iromli <isman.firmansyah@gmail.com> * fix: resolve required password files Signed-off-by: iromli <isman.firmansyah@gmail.com> * fix: pre-populate LDAP bindDN Signed-off-by: iromli <isman.firmansyah@gmail.com> * fix: remove duplicated jansAccessTknSigAlg attribute Signed-off-by: iromli <isman.firmansyah@gmail.com> * chore: clarify local secrets and configmaps will be excluded if configuration.json is missing Signed-off-by: iromli <isman.firmansyah@gmail.com> * refactor: remove unused backward-compat Signed-off-by: iromli <isman.firmansyah@gmail.com> * refactor: bootstrap Vault RoleID and SecretID (if required) Signed-off-by: iromli <isman.firmansyah@gmail.com> --------- Signed-off-by: iromli <isman.firmansyah@gmail.com> Signed-off-by: Isman Firmansyah <iromli@users.noreply.github.com> Former-commit-id: fc9f04f

feat(jans-pycloudlib): add support reading configuration from file

2ac918b

Signed-off-by: iromli <isman.firmansyah@gmail.com>

mo-auto added comp-jans-pycloudlib kind-feature Issue or PR is a new feature request labels Jul 26, 2024

github-advanced-security bot found potential problems Jul 26, 2024

View reviewed changes

jans-pycloudlib/jans/pycloudlib/manager.py Fixed Show fixed Hide fixed

jans-pycloudlib/jans/pycloudlib/manager.py Fixed Show fixed Hide fixed

tests(jans-pycloudlib): adjust testcases

8104fa0

Signed-off-by: iromli <isman.firmansyah@gmail.com>

iromli self-assigned this Jul 29, 2024

iromli and others added 9 commits July 30, 2024 02:48

feat(jans-pycloudlib): add CLI command to generate configuration spec

1be8af9

Signed-off-by: iromli <isman.firmansyah@gmail.com>

refactor: simplified configmaps and secrets adapters

95e0f04

Signed-off-by: iromli <isman.firmansyah@gmail.com>

fix: handle missing params

e38839d

Signed-off-by: iromli <isman.firmansyah@gmail.com>

fix: add backward-compat for configuration and dump files

5243d37

Signed-off-by: iromli <isman.firmansyah@gmail.com>

Merge branch 'main' into cn-pycloudlib-proxy

4a43d2b

Merge branch 'main' into cn-pycloudlib-proxy

e9131b1

test(jans-pycloudlib): fix transform_data testcase

2cb64b5

Signed-off-by: iromli <isman.firmansyah@gmail.com>

Merge branch 'main' into cn-pycloudlib-proxy

c21f0b6

Merge branch 'main' into cn-pycloudlib-proxy

1b949fa

iromli and others added 11 commits August 20, 2024 23:30

refactor: handle mounted password files

4120ae2

Signed-off-by: iromli <isman.firmansyah@gmail.com>

refactor: handle mounted files in OCI images

d60a048

Signed-off-by: iromli <isman.firmansyah@gmail.com>

fix: ensure couchbase password files are exist

ae91c11

Signed-off-by: iromli <isman.firmansyah@gmail.com>

refactor: generate opendj.pkcs12 on-the-fly

9cd9868

Signed-off-by: iromli <isman.firmansyah@gmail.com>

fix: resolve required password files

1000e97

Signed-off-by: iromli <isman.firmansyah@gmail.com>

fix: pre-populate LDAP bindDN

03f07fd

Signed-off-by: iromli <isman.firmansyah@gmail.com>

fix: remove duplicated jansAccessTknSigAlg attribute

6b5c32b

Signed-off-by: iromli <isman.firmansyah@gmail.com>

chore: clarify local secrets and configmaps will be excluded if confi…

36ef0b4

…guration.json is missing Signed-off-by: iromli <isman.firmansyah@gmail.com>

Merge branch 'main' into cn-pycloudlib-proxy

4e3bfee

refactor: remove unused backward-compat

697339c

Signed-off-by: iromli <isman.firmansyah@gmail.com>

refactor: bootstrap Vault RoleID and SecretID (if required)

f11ab23

Signed-off-by: iromli <isman.firmansyah@gmail.com>

Merge branch 'main' into cn-pycloudlib-proxy

c5b70dc

Signed-off-by: Isman Firmansyah <iromli@users.noreply.github.com>

iromli marked this pull request as ready for review August 27, 2024 15:18

iromli requested a review from moabu as a code owner August 27, 2024 15:18

mo-auto added comp-docker-jans-auth-server comp-docker-jans-link comp-docker-jans-certmanager comp-docker-jans-config-api comp-docker-jans-configurator comp-docker-jans-fido2 comp-docker-jans-persistence-loader comp-docker-jans-scim labels Aug 27, 2024

moabu approved these changes Aug 27, 2024

View reviewed changes

moabu merged commit fc9f04f into main Aug 27, 2024
18 of 19 checks passed

moabu deleted the cn-pycloudlib-proxy branch August 27, 2024 17:06

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(jans-pycloudlib): add support reading configuration from file #9037

feat(jans-pycloudlib): add support reading configuration from file #9037

iromli commented Jul 26, 2024 •

edited

Loading

dryrunsecurity bot commented Jul 26, 2024 •

edited

Loading

sonarqubecloud bot commented Aug 20, 2024

sonarqubecloud bot commented Aug 20, 2024

sonarqubecloud bot commented Aug 20, 2024

sonarqubecloud bot commented Aug 26, 2024

sonarqubecloud bot commented Aug 27, 2024

sonarqubecloud bot commented Aug 27, 2024

sonarqubecloud bot commented Aug 27, 2024

sonarqubecloud bot commented Aug 27, 2024

sonarqubecloud bot commented Aug 27, 2024

sonarqubecloud bot commented Aug 27, 2024

sonarqubecloud bot commented Aug 27, 2024

feat(jans-pycloudlib): add support reading configuration from file #9037

feat(jans-pycloudlib): add support reading configuration from file #9037

Conversation

iromli commented Jul 26, 2024 • edited Loading

Prepare

Description

Target issue

Implementation Details

Test and Document the changes

dryrunsecurity bot commented Jul 26, 2024 • edited Loading

DryRun Security Summary

Code Analysis

Riskiness

sonarqubecloud bot commented Aug 20, 2024

Quality Gate passed for 'agama parent'

sonarqubecloud bot commented Aug 20, 2024

Quality Gate passed for 'keycloak-integration-parent'

sonarqubecloud bot commented Aug 20, 2024

Quality Gate passed for 'orm'

sonarqubecloud bot commented Aug 26, 2024

Quality Gate passed for 'jans-pycloudlib'

sonarqubecloud bot commented Aug 27, 2024

Quality Gate passed for 'jans-cli'

sonarqubecloud bot commented Aug 27, 2024

Quality Gate passed for 'Fido2 API'

sonarqubecloud bot commented Aug 27, 2024

Quality Gate passed for 'jans-config-api-parent'

sonarqubecloud bot commented Aug 27, 2024

Quality Gate passed for 'Jans-Keycloak-Link'

sonarqubecloud bot commented Aug 27, 2024

Quality Gate passed for 'jans-core'

sonarqubecloud bot commented Aug 27, 2024

Quality Gate passed for 'jans-linux-setup'

sonarqubecloud bot commented Aug 27, 2024

Quality Gate passed for 'SCIM API'

iromli commented Jul 26, 2024 •

edited

Loading

dryrunsecurity bot commented Jul 26, 2024 •

edited

Loading