v0.17.0

Release 0.17.0 of simtools, the Simulation tools and applications for the CTA Observatory, includes the following main changes:

Bugfixes

• Fix data type of 'dsum_threshold' parameter. (#1557)
• Add missing neutrons to eventio particle IDs. (#1584)
• Fix suffix for sim_telarray output files: no correctly simtel.zst (was .zst only) (#1588)

Documentation

• Change documentation theme to pydata and add version switcher. (#1560)
• Update documentation on how to run jobs using HT Condor. (#1563)

New Features

• Add pixel layout plotting. (#1541)
• Enable docker-image generation for vector-optimized CORSIKA 7.8. (#1550)
• Add the sequential option to simulations using multipipe. With this option the CORSIKA and sim_telarray instances run in sequential order as far as possible. This mode is useful particularly on the grid, where typically we request a single core per job. (#1561)
• Add reading and conversion of primary_id from sim_telarray output. (#1570)
• Update simtools-generate-simtel-event-data to use astropy tables. (#1573)
• Generalize table reading and writing. Add application to merge tables. (#1580)
• Write reduced event data file as part of the cleanup stage of the simtools-simulate-prod application. (#1588)
• Use patches for optimized CORSIKA from gitlab repository instead of cloud download. (#1589)
• New utility simtools-verify-simulation-model-production-tables to verify completeness of model parameter repository. (#1592)
• Add custom copilot instructions to support AI used by developers. (#1594)
• Add new application simtools-submit-array-layouts to submit new array-layouts definition and validate the telescopes listed. (#1596)
• Improve layout plotting; adding plotting of background layout. Add plotting of all layouts. (#1597)
• Improve printout for assertion error for testing. (#1598)
• Add production version and build options to sim_telarrary metadata. (#1601)

Maintenance

• Add unit tests to increase coverage in interpolation handler. (#1504)
• Docker file improvements: linter application; bugfix in extra flags for optimized CORSIKA compilation. (#1544)
• Include stdout and stderr in error message of integration test. (#1555)
• Add pytest-retry package for 'flaky test'. (#1558)
• Reduce number of log-messages (INFO) for uploaded model parameters to DB. (#1559)
• Replace the "run_number_start" argument with "run_number_offset" (clearer) and re-introduce the "run_number" argument to allow specifying a run number (to which the offset is added). (#1562)
• Use telescope position files for setup in illuminator application and set correct obs level. (#1566)
• Improve the names of the various simulation output files. They now follow the current convention in current productions, they contain a bit more information (e.g., the model version where it wasn't there before), and they are more consistent with each other. (#1568)
• Improved interface and interpolation routines to derive production configuration. This creates a pipeline for lookup table generation, production grid generation, creating a grid with limits, which is then further complemented by the production statistics per grid point. (#1569)
• Fix unit test to run with astropy 7.1 (and earlier versions) (see #1574). (#1575)
• Simplified Sonar setup by moving properties into GitHub workflow. (#1578)
• Introduce simplified Codecov configuration. (#1579)
• Copy the CORSIKA log in binary mode instead of text mode. (#1582)
• Protect against non-unicode characters in CORSIKA log file. (#1583)
• Remove obsolete code on submission systems in JobManager. (#1586)
• Change dummy trigger thresholds to easier-to-spot values. (#1591)
• Fix duplication of some sim_telarray test files in './tests/resources' and corresponding test fixtures. (#1593)

v0.16.0

Release 0.16.0 of simtools, the Simulation tools and applications for the CTA Observatory, includes the following main changes:

Documentation

• Add a section on mocking to the developer documentation. (#1516)
• Improve model parameter description (schemas, versions) in documentation. (#1524)

New Features

• Add grid generation production application to generate the grid points for Corsika simulations, with energy thresholds, viewcone and radius interpolation. (#1482)
• Add support for multiple versions and the multipipe functionality of sim_telarray (#1496)
• Add validation metadata written by sim_telarray with array model parameters. (#1505)
• Improve robustness of sim_telarray configuration by adding an invalid (dummy) telescope configuration. (#1509)
• Adding functionality for producing simulation configuration reports. (#1510)
• Add random seeds from file for instrument configuration. (#1511)
• Adapt docker files and building workflows to CORSIKA 7.8000. (#1512)
• Add sim_telarray random seed for instrument configuration to metadata (and metadata tests). (#1515)
• Integration test with multipipe and support for packing all output files in grid productions. (#1520)
• Add versioning to model parameter schema files. Implement example for corsika_starting_grammage. (#1523)
• Added an application to produce calibration device reports. (#1530)
• Allow corsika_starting_grammage to be telescope-type dependent. (#1539)

Maintenance

• Separate TelescopeModel and SiteModel for improved readability in sim_telarray configuration writer. (#1489)
• Fix licence definition in pyproject.toml to follow PEP 639. (#1493)
• Improve consistency of spelling of sim_telarray. (#1494)
• Remove test condition in simulator light emission and mock configuration file access. (#1495)
• Refactoring of event scaling application and moving most of the functionality into a new module. (#1498)
• Renaming of event scaling methods to production statistics derivation methods. (#1502)
• Improved naming and docstrings in simulation runners. (#1519)
• Remove duplication of reading yaml/json files in simtools.utils.general. (#1522)
• Remove duplication in make_run_command in corsika_simtel_runner and simulator_array. (#1525)
• Collect pyeventio related functions in new module simtel_io_file_info. (#1531)
• Add new field model_parameter_schema_version to model parameter test files. (#1535)

Simulation model

• Change list of applicable array elements and class for pedestal_events model parameter. (#1490)

v0.15.0

Release 0.15.0 of simtools, the Simulation tools and applications for the CTA Observatory, includes the following main changes:

Maintenance

• Skip testing of DB upload for simpipe integration tests. (#1488)

v0.14.0

Release 0.14.0 of simtools, the Simulation tools and applications for the CTA Observatory, includes the following main changes:

Bugfixes

• Fix integration tests uploading values / files to sandbox DB. (#1453)
• Fix building of production images from branches other than main. (#1467)
• Add missing parameter definition in model parameter schema. (#1470)
• Primary degraded map is a parameter applied for all telescope types. (#1475)
• Fixes a bug in the selection of the triggered events by telescope IDs for the limits calculation. (#1483)
• Fix how integration tests are marked for use cases (used in SimPipe AIV). (#1484)

New Features

• Added functionality to automate report generation for array elements and parameters. (#1436)
• Add full set of array pointings to reduced mc event data file if available in the mc header. (#1439)
• Add fitting of afterpulse spectrum to derive single pe application. (#1446)
• Run scheduled nightly integration and unit tests on production DB. (#1454)
• Generalize extraction of MC event tree from sim_telarrary files; generic writer and reader. (#1458)
• Add functionality to generate reports for site-specific parameters. (#1459)
• Allow to skip integration tests for production DB. (#1460)
• Add an application to print the versions of simtools, the DB, sim_telarray, and CORSIKA (simtools-print-version). (#1461)
• Use files to pass filepaths for simtel event data files and telescope IDs in corsika limits derivation tool. (#1464)
• Retrieve CTAO layout definitions. Merge with model parameter value and write to disk. (#1465)
• Add tests to ensure that model table files uploaded to DB follow Unicode. (#1473)
• Add writing of metadata into sim_telarray configuration files. (#1474)
• Update model parameter definitions to reflect those from most recent sim_telarray release. (#1476)

Maintenance

• Remove docker prod image generation; documentation update and improved workflow naming. (#1447)
• Change requirement of arguments in event scaling application and improve documentation. (#1449)
• Refactor light emission application and move logic to helper functions. (#1450)
• Use consistently numpy dtypes for model parameters. (#1451)
• Remove h5py dependency. (#1456)
• Store arrays in model parameters as arrays and not as strings. (#1466)

v0.13.0

Release 0.13.0 of simtools, the Simulation tools and applications for the CTA Observatory, includes the following main changes:

Bugfixes

• Fix null values in YAML-style model parameter schema files. (#1437)

New Features

• Improvements for single pe setting workflow including plotting and metadata handling. (#1398)
• Add an application that allows to extract array and shower data and to save them in hdf5 format for the later calculation of production configuration limits. (#1402)
• Add integration tests to production image generation. (#1420)
• Improve reading of single parameter and files from database. Add plotting of tabular data using parameter_version. (#1426)
• Improved plotting of tabular data; adding error bars and schema validation for plotting configuration. (#1429)
• Add function to get all model parameter data for all model versions. (#1432)

Maintenance

• Run unit and integration test on database setup in local test environment. (#1424)
• Remove tests from pypi package. Remove obsolete __init__.py files. (#1430)
• Change base python version from 3.11 to 3.12. (#1433)
• Update exporting of sim_telarray model to simtools taking into account the updated file naming (included parameter versions). (#1438)
• Robust getting of username in metadata collector in case no user is defined on system level. (#1442)

v0.12.0

Release 0.12.0 of the Simulation tools and applications for the CTA Observatory includes the following main changes:

0.12.0 - 2025-03-11

Bugfixes

• Bugfix in database-upload related string to list conversion. (#1423)
• Fix setting of db_api_authentication_database through env variable. (#1425)

v0.11.0

Release 0.11.0 of the Simulation tools and applications for the CTA Observatory includes the following main changes:

Bugfixes

• Fix yamllint options in pre-commit (was essentially switched off). Fix syntax in many YAML files. (#1404)
• Fix date writing to ensure RFC 3339-conform date format. (#1411)
• Fix reading of combined units from model parameter unit string. (#1415)

New Features

• Adding two new applications for generating model parameter documentation reports for a given array element either for a given production, or across multiple productions. (#1185)
• Add building of optimized corsika binaries (e.g., avx2, avx512) and building of images for different corsika versions. (#1355)
• Add a simtool to derive limits for energy, viewcone and radius for a simulation. (#1356)
• Add generic simtools-run-application to run one or several simtools using a single configuration file. (#1379)
• Add reading of database collections from model parameter schemas. (#1380)
• Add module for simtools dependency management. Allows to retrieve e.g. sim_telarray and CORSIKA versions. (#1383)
• Expanded functionality of application configuration workflows based on the integration test configurations. (#1389)
• Set user in docker images and add possibility to add user information from env/command line. (#1400)
• Allow to mark integration tests in their use case config with a corresponding CTAO use case or requirement ID. (#1405)
• Run check for changelog entry only on PRs ready for review. (#1416)

Maintenance

• Maintenance in production configuration removing not needed flags and only use metric files. (#1376)
• Rename derive-limits application to production-derive-limits. (#1378)
• Add YAML file with build opts into CORSIKA/sim_telarray images. (#1382)
• Add unit tests for version.py. (#1384)
• Improve function naming and unit test coverage in simtools.utils.names. (#1390)
• Change checks for software updates from weekly to monthly. (#1394)
• Remove duplication in converting sim_telarray string-lists (values, units) to python lists. (#1395)
• Improve reading of design models; refactoring and improved docstrings for simtools.utils.names. (#1396)

Simulation model

• Introduce MSTx-NectarCam and MSTx-FlashCam design models for MSTs (allow both sites). (#1362)

v0.10.0

Release 0.10.0 of the Simulation tools and applications for the CTA Observatory includes the following main changes:

Bugfixes

• Schema validation bugfixes. (#1330)

New Features

• Add a simtool to derive (normalize) the single p.e. amplitude spectrum using the sim_telarray tool norm_spe. (#1299)
• Add validation of simtools-generated configuration files for sim_telarray with reference files. (#1322)
• Add schema for fake mirror list. (#1338)
• Add additional queries to db handler to retrieve model versions etc. (#1349)
• Add fixed database version for testing (instead of LATEST). (#1365)

Maintenance

• Major restructering of database routines plus introduction of new simulation model. (#1316)
• Improve testing of results and allow comparison of lists of floats in json/YAML files. (#1319)
• Remove database sandboxes . (#1336)

What's Changed

• Fix label setting in validate_camera_efficiency by @GernotMaier in #1320
• Update pre-commit hooks by @github-actions in #1325
• Clear default sim_telarray cfg directories from all calls by @orelgueta in #1324
• Allow for validation of simtools-generated configuration files for sim_telarray with reference files. by @GernotMaier in #1322
• Schema validation bug fixes. by @GernotMaier in #1330
• Reference model corrections (new db simulation model) by @GernotMaier in #1331
• Set trigger current limit to 20 uA by @GernotMaier in #1334
• Correct branch to main for simulation model upload by @GernotMaier in #1335
• Improve testing of results and allow comparision of lists of floats in json/yaml files by @GernotMaier in #1319
• Add schema for fake mirror list by @GernotMaier in #1338
• Remove sandboxes by @GernotMaier in #1336
• Simulation model refactoring with refactoring of db_handler. by @GernotMaier in #1310
• Update testeff command for SSTs/SCTs to use fake mirror list by @GernotMaier in #1339
• Update pre-commit hooks by @github-actions in #1341
• Replace model parameter default schema versions by value read from model_parameter.metaschema.yml by @GernotMaier in #1332
• Remove duplication for schema validation plus some simplification by @GernotMaier in #1347
• Minor code / maintenance updates to production configuration setup. by @GernotMaier in #1345
• Production configuration schema and validation by @GernotMaier in #1348
• Add additional queries to db handler to retrieve model versions etc. by @GernotMaier in #1349
• Maintenance PR for simtel io histogram(s) modules. by @GernotMaier in #1352
• Update pre-commit hooks by @github-actions in #1358
• Schema typos and URL fixes by @GernotMaier in #1359
• Missing sphinx documentation by @GernotMaier in #1363
• Simtools for single pe derivation by @GernotMaier in #1299
• Add fixed database version for testing by @GernotMaier in #1365
• Add input validation for single pe tool. by @GernotMaier in #1350
• Add bin edges to hdf5 by @tobiaskleiner in #1354
• Make sure that Changelog entries are not forgotten. by @GernotMaier in #1368
• Update pre-commit hooks by @github-actions in #1369
• Validate meta_schema_url by @GernotMaier in #1366
• Prepare simtools release 0.10.0 by @GernotMaier in #1370

Full Changelog: v0.9.0...v0.10.0

v0.9.0

Release 0.9.0 of the Simulation tools and applications for the CTA Observatory includes the following main changes:

Bugfixes

• Add correct handling of return codes from simulation software to prevent reports of success even though a job failed (simulate_prod) (#1289)
• Fix setting of activity:end time in the metadata collector. (#1291)

New Features

• Add an application to plot tables from a file (from file system or for a model parameter file downloaded from the DB). (#1267)
• Enhancements to file testing within integration tests. (#1279)
• Add a tool to prepare job submission with simulate_prod for a HTCondor system. (#1290)
• Remove functionality in db_handler to read from simulation model repository. (#1306)

Maintenance

• Change dependency from ctapipe to ctapipe-base for ctapipe v0.23 (#1247)
• Remove pytest dependent from user installation. (#1286)

What's Changed

• Change in optimization algorithm for the derivation of the random reflection angle parameter. by @GernotMaier in #1275
• Use astropy-base package in conda environment. by @GernotMaier in #1272
• CTAO CR spectra package now available from conda by @GernotMaier in #1277
• Update pre-commit hooks by @github-actions in #1278
• Update pre-commit hooks by @github-actions in #1282
• Improve db configuration checks by @GernotMaier in #1281
• Replace os.system by subprocess.run and Pathlib commands. by @GernotMaier in #1283
• Improvements for the job submission using HT Condor by @GernotMaier in #1279
• Move test_commandline_tools as test after simtools-prod container generation. by @GernotMaier in #1285
• Add a simple tool to prepare job submission with simulate_prod for a HTCondor system. by @GernotMaier in #1290
• Set correct activity:end in metadata by @GernotMaier in #1291
• Handle return codes correctly for simulation productions by @GernotMaier in #1289
• Add corsika log file collection for --pack_for_grid_register in simtools-simulate-prod by @GernotMaier in #1294
• Bump SonarSource/sonarqube-scan-action from 4.1.0 to 4.2.1 by @dependabot in #1296
• Update pre-commit hooks by @github-actions in #1295
• Fix failing unit tests in job_manager by @GernotMaier in #1297
• Update pre-commit hooks by @github-actions in #1298
• Remove pytest dependent from user installation. by @GernotMaier in #1286
• Optimize scipy imports in camera module by @GernotMaier in #1292
• Use-friendly naming of application documentation by @GernotMaier in #1300
• Fix validation of model parameters using schemas. by @GernotMaier in #1302
• Update pre-commit hooks by @github-actions in #1303
• Remove functionality in db_handler to read from simulation model repository by @GernotMaier in #1306
• Remove obsolute functions from db_handler by @GernotMaier in #1307
• Simplify db_handler.add_new_parameter and add parameter dict validation before writing by @GernotMaier in #1308
• Improvements to handle different versions of schemas. by @GernotMaier in #1309
• Update pre-commit hooks by @github-actions in #1311
• Simple plotting application for tabular data by @GernotMaier in #1267
• Preparation of data validation routines for db restructering by @GernotMaier in #1312
• Require explicit selection of model_version as command line parameter by @GernotMaier in #1313
• Fix site in integration test by @GernotMaier in #1314

Full Changelog: v0.8.2...v0.9.0

v0.8.2

Simulation tools and applications for the CTA Observatory.

Release 0.8.2 is a maintenance release with no changes in functionality. Main changes a new layout for the simtools code repository (src-layout).

What's Changed

• Update README.md by @tobiaskleiner in #1258
• type hints for models by @tobiaskleiner in #1253
• Update pre-commit hooks by @github-actions in #1262
• Improve help and printout for --telescope_model_file command line option. by @GernotMaier in #1263
• Bump SonarSource/sonarqube-scan-action from 4.0.0 to 4.1.0 by @dependabot in #1265
• Change simtools package layout from 'flat' to 'src'. by @GernotMaier in #1264
• Change schema url to scr-layout by @GernotMaier in #1268
• Changes related to src-layout move. by @GernotMaier in #1270

Full Changelog: v0.8.1...v0.8.2