ENH: add Makefile information helper #124

klauer · 2022-03-30T00:01:39Z

Description

Start of a tool to help inspect Makefile information
Currently allows for inspection of all makefile variables and the default make target
Splits out simple parse to whatrecord.parse in order to make it more accessible. External imports relying on .bin.parse.parse should remain functional.

This method actually reveals a ton of information about the build when used with EPICS makefiles. It's a pretty interesting exercise, and I think we may be able to leverage this for more than I initially thought:

Variables

    "EPICSDEV": "/reg/g/pcds/epics-dev",
    "EPICSPROD": "/reg/g/pcds/epics",
    "EPICS_BASE": "/cds/group/pcds/epics/base/R7.0.3.1-2.0",
    "EPICS_BASE_HOST_BIN": "/cds/group/pcds/epics/base/R7.0.3.1-2.0/bin/linux-x86_64",
    "EPICS_BASE_HOST_LIB": "/cds/group/pcds/epics/base/R7.0.3.1-2.0/lib/linux-x86_64",
    "EPICS_BASE_HOST_LIBS": "ca Com",
    "EPICS_BASE_IOC_LIBS": "dbRecStd dbCore ca Com",
    "EPICS_BASE_PVA_CORE_LIBS": "pvAccessIOC pvAccessCA pvAccess pvData",
    "EPICS_BASE_VERSION": "7.0.5.0",
    "EPICS_CA_AUTO_ADDR_LIST": "YES",
    "EPICS_CA_BEACON_PERIOD": "15.0",
    "EPICS_CA_CONN_TMO": "30.0",
    "EPICS_CA_DEVELOPMENT_FLAG": "0",
    "EPICS_CA_MAINTENANCE_VERSION": "5",
    "EPICS_CA_MAJOR_VERSION": "4",
    "EPICS_CA_MAX_ARRAY_BYTES": "40000000",
    "EPICS_CA_MAX_SEARCH_PERIOD": "300",
    "EPICS_CA_MINOR_VERSION": "13",
    "EPICS_CA_REPEATER_PORT": "5065",
    "EPICS_CA_SERVER_PORT": "5064",
    "EPICS_DATABASE": "/cds/group/pcds/epics/base/R7.0.3.1-2.0",
    "EPICS_DATABASE_DEVELOPMENT_FLAG": "0",
    "EPICS_DATABASE_HOST_BIN": "/cds/group/pcds/epics/base/R7.0.3.1-2.0/bin/linux-x86_64",
    "EPICS_DATABASE_MAINTENANCE_VERSION": "5",
    "EPICS_DATABASE_MAJOR_VERSION": "3",
    "EPICS_DATABASE_MINOR_VERSION": "17",
    "EPICS_DEV_SNAPSHOT": "",
    "EPICS_EXTENSIONS": "/reg/g/pcds/epics/extensions/R0.2.0",
    "EPICS_HOST_ARCH": "linux-x86_64",
    "EPICS_LIBCOM_DEVELOPMENT_FLAG": "0",
    "EPICS_LIBCOM_MAINTENANCE_VERSION": "6",
    "EPICS_LIBCOM_MAJOR_VERSION": "3",
    "EPICS_LIBCOM_MINOR_VERSION": "17",
    "EPICS_MODIFICATION": "3",
    "EPICS_MODULES": "/cds/group/pcds/epics/R7.0.3.1-2.0/modules",
    "EPICS_NTYPES_DEVELOPMENT_FLAG": "0",
    "EPICS_NTYPES_MAINTENANCE_VERSION": "1",
    "EPICS_NTYPES_MAJOR_VERSION": "6",
    "EPICS_NTYPES_MINOR_VERSION": "0",
    "EPICS_PATCH_LEVEL": "1",
    "EPICS_PATCH_VSTRING": ".1",
    "EPICS_PVACLIENT_DEVELOPMENT_FLAG": "0",
    "EPICS_PVACLIENT_MAINTENANCE_VERSION": "0",
    "EPICS_PVACLIENT_MAJOR_VERSION": "4",
    "EPICS_PVACLIENT_MINOR_VERSION": "7",
    "EPICS_PVA_AUTO_ADDR_LIST": "YES",
    "EPICS_PVA_BROADCAST_PORT": "5076",
    "EPICS_PVA_DEVELOPMENT_FLAG": "0",
    "EPICS_PVA_MAINTENANCE_VERSION": "0",
    "EPICS_PVA_MAJOR_VERSION": "7",
    "EPICS_PVA_MINOR_VERSION": "1",
    "EPICS_PVA_SERVER_PORT": "5075",
    "EPICS_PVDATABASE_DEVELOPMENT_FLAG": "0",
    "EPICS_PVDATABASE_MAINTENANCE_VERSION": "1",
    "EPICS_PVDATABASE_MAJOR_VERSION": "4",
    "EPICS_PVDATABASE_MINOR_VERSION": "5",
    "EPICS_PVD_DEVELOPMENT_FLAG": "0",
    "EPICS_PVD_MAINTENANCE_VERSION": "1",
    "EPICS_PVD_MAJOR_VERSION": "8",
    "EPICS_PVD_MINOR_VERSION": "0",
    "EPICS_QSRV_ABI_MAJOR_VERSION": "1",
    "EPICS_QSRV_ABI_MINOR_VERSION": "1",
    "EPICS_QSRV_DEVELOPMENT_FLAG": "0",
    "EPICS_QSRV_MAINTENANCE_VERSION": "2",
    "EPICS_QSRV_MAJOR_VERSION": "1",
    "EPICS_QSRV_MINOR_VERSION": "2",
    "EPICS_REPO": "file:///afs/slac/g/pcds/vol2/svn/pcds",
    "EPICS_REVISION": "0",
    "EPICS_SETUP": "/reg/g/pcds/setup",
    "EPICS_SHORT_VERSION": "7.0.3.1",
    "EPICS_SITE_TOP": "/cds/group/pcds/epics",
    "EPICS_SITE_VERSION": "2.0.1",
    "EPICS_SITE_VSTRING": "-2.0.1",
    "EPICS_VERSION": "7",
    "EPICS_VERSION_NUMBER": "7.0.3.1-2.0.1",
    "EPICS_VERSION_STRING": "\"EPICS Version 7.0.3.1-2.0.1\"",

...

    "INSTALL": "perl -CSD /cds/group/pcds/epics/base/R7.0.3.1-2.0/bin/linux-x86_64/installEpics.pl -q",
    "INSTALLS_CFG": "",
    "INSTALL_ADL": "./adl",
    "INSTALL_ALH": "./alh",
    "INSTALL_ARCHIVE": "./archive",
    "INSTALL_AUTOSAVE": "./autosave",
    "INSTALL_CFG": "./cfg",
    "INSTALL_CMD": "./db",
    "INSTALL_CONFIG": "./configure",
    "INSTALL_DB": "./db",
    "INSTALL_DBD": "./dbd",
    "INSTALL_DOC": "./doc",
    "INSTALL_EDL": "./edl",
    "INSTALL_HOST_BIN": "./bin/linux-x86_64",
    "INSTALL_HOST_LIB": "./lib/linux-x86_64",
    "INSTALL_HTML": "./html",
    "INSTALL_INCLUDE": "./include",
    "INSTALL_LIBRARY": "perl -CSD /cds/group/pcds/epics/base/R7.0.3.1-2.0/bin/linux-x86_64/installEpics.pl -q",
    "INSTALL_LOCATION": ".",
    "INSTALL_LOCATION_BIN": "./bin",
    "INSTALL_LOCATION_LIB": "./lib",
    "INSTALL_PERL_MODULES": "./lib/perl",
    "INSTALL_PKGCONFIG": "./lib/pkgconfig",
    "INSTALL_PRODUCT": "perl -CSD /cds/group/pcds/epics/base/R7.0.3.1-2.0/bin/linux-x86_64/installEpics.pl -q",
    "INSTALL_PROTO": "./db",
    "INSTALL_RESTORE": "./restore",
    "INSTALL_TEMPLATES": "./templates",

...
    "RELEASE_CFG_CONFIGS": "             /cds/group/pcds/epics/base/R7.0.3.1-2.0/cfg/CONFIG_CA_VERSION /cds/group/pcds/epics/base/R7.0.3.1-2.0/cfg/CONFIG_NTYPES_VERSION /cds/group/pcds/epics/base/R7.0.3.1-2.0/cfg/CONFIG_DATABASE_VERSION /cds/group/pcds/epics/base/R7.0.3.1-2.0/cfg/CONFIG_PVACLIENT_VERSION /cds/group/pcds/epics/base/R7.0.3.1-2.0/cfg/CONFIG_QSRV_VERSION /cds/group/pcds/epics/base/R7.0.3.1-2.0/cfg/CONFIG_PVACCESS_MODULE /cds/group/pcds/epics/base/R7.0.3.1-2.0/cfg/CONFIG_PVDATA_VERSION /cds/group/pcds/epics/base/R7.0.3.1-2.0/cfg/CONFIG_PVDATABASE_VERSION /cds/group/pcds/epics/base/R7.0.3.1-2.0/cfg/CONFIG_PVACCESS_VERSION /cds/group/pcds/epics/base/R7.0.3.1-2.0/cfg/CONFIG_CA_MODULE /cds/group/pcds/epics/base/R7.0.3.1-2.0/cfg/CONFIG_LIBCOM_MODULE /cds/group/pcds/epics/base/R7.0.3.1-2.0/cfg/CONFIG_DATABASE_MODULE /cds/group/pcds/epics/base/R7.0.3.1-2.0/cfg/CONFIG_LIBCOM_VERSION        ",
    "RELEASE_CFG_DIR_RULES": "                     ",
    "RELEASE_CFG_TOP_RULES": "                     ",
    "RELEASE_TOPS": "BASE_MODULE_VERSION EPICS_SITE_TOP EPICS_MODULES PSPKG_ROOT EPICS_BASE AUTOSAVE_MODULE_VERSION IOCADMIN_MODULE_VERSION CAPUTLOG_MODULE_VERSION IOCADMIN AUTOSAVE CAPUTLOG",
    "RELEASE_TOPS_REVERSE": "CAPUTLOG AUTOSAVE IOCADMIN CAPUTLOG_MODULE_VERSION IOCADMIN_MODULE_VERSION AUTOSAVE_MODULE_VERSION EPICS_BASE PSPKG_ROOT EPICS_MODULES EPICS_SITE_TOP BASE_MODULE_VERSION",

...


    "INSTALL": "perl -CSD /cds/group/pcds/epics/base/R7.0.3.1-2.0/bin/linux-x86_64/installEpics.pl -q",
    "INSTALLS_CFG": "",
    "INSTALL_ADL": "./adl",
    "INSTALL_ALH": "./alh",
    "INSTALL_ARCHIVE": "./archive",
    "INSTALL_AUTOSAVE": "./autosave",
    "INSTALL_CFG": "./cfg",
    "INSTALL_CMD": "./db",
    "INSTALL_CONFIG": "./configure",
    "INSTALL_DB": "./db",
    "INSTALL_DBD": "./dbd",
    "INSTALL_DOC": "./doc",
    "INSTALL_EDL": "./edl",
    "INSTALL_HOST_BIN": "./bin/linux-x86_64",
    "INSTALL_HOST_LIB": "./lib/linux-x86_64",
    "INSTALL_HTML": "./html",
    "INSTALL_INCLUDE": "./include",
    "INSTALL_LIBRARY": "perl -CSD /cds/group/pcds/epics/base/R7.0.3.1-2.0/bin/linux-x86_64/installEpics.pl -q",
    "INSTALL_LOCATION": ".",
    "INSTALL_LOCATION_BIN": "./bin",
    "INSTALL_LOCATION_LIB": "./lib",
    "INSTALL_PERL_MODULES": "./lib/perl",
    "INSTALL_PKGCONFIG": "./lib/pkgconfig",
    "INSTALL_PRODUCT": "perl -CSD /cds/group/pcds/epics/base/R7.0.3.1-2.0/bin/linux-x86_64/installEpics.pl -q",
    "INSTALL_PROTO": "./db",
    "INSTALL_RESTORE": "./restore",
    "INSTALL_TEMPLATES": "./templates",

...

    "MAKEFILE_LIST": " /tmp/Gm9iER6U configure/CONFIG configure/RELEASE RELEASE_SITE /cds/group/pcds/epics/base/R7.0.3.1-2.0/configure/CONFIG /cds/group/pcds/epics/base/R7.0.3.1-2.0/configure/RELEASE /cds/group/pcds/epics/base/R7.0.3.1-2.0/configure/CONFIG_COMMON /cds/group/pcds/epics/base/R7.0.3.1-2.0/configure/CONFIG_FILE_TYPE /cds/group/pcds/epics/base/R7.0.3.1-2.0/configure/CONFIG_BASE /cds/group/pcds/epics/base/R7.0.3.1-2.0/configure/CONFIG_SITE /cds/group/pcds/epics/base/R7.0.3.1-2.0/configure/CONFIG_BASE_VERSION /cds/group/pcds/epics/base/R7.0.3.1-2.0/configure/os/CONFIG.linux-x86_64.Common /cds/group/pcds/epics/base/R7.0.3.1-2.0/configure/os/CONFIG.UnixCommon.Common /cds/group/pcds/epics/base/R7.0.3.1-2.0/configure/os/CONFIG_SITE.linux-x86_64.Common /cds/group/pcds/epics/base/R7.0.3.1-2.0/cfg/CONFIG_PVDATA_VERSION /cds/group/pcds/epics/base/R7.0.3.1-2.0/cfg/CONFIG_CA_VERSION /cds/group/pcds/epics/base/R7.0.3.1-2.0/cfg/CONFIG_LIBCOM_MODULE /cds/group/pcds/epics/base/R7.0.3.1-2.0/cfg/CONFIG_PVDATABASE_VERSION /cds/group/pcds/epics/base/R7.0.3.1-2.0/cfg/CONFIG_DATABASE_MODULE /cds/group/pcds/epics/base/R7.0.3.1-2.0/cfg/CONFIG_NTYPES_VERSION /cds/group/pcds/epics/base/R7.0.3.1-2.0/cfg/CONFIG_PVACCESS_VERSION /cds/group/pcds/epics/base/R7.0.3.1-2.0/cfg/CONFIG_CA_MODULE /cds/group/pcds/epics/base/R7.0.3.1-2.0/cfg/CONFIG_LIBCOM_VERSION /cds/group/pcds/epics/base/R7.0.3.1-2.0/cfg/CONFIG_DATABASE_VERSION /cds/group/pcds/epics/base/R7.0.3.1-2.0/cfg/CONFIG_PVACLIENT_VERSION /cds/group/pcds/epics/base/R7.0.3.1-2.0/cfg/CONFIG_QSRV_VERSION /cds/group/pcds/epics/base/R7.0.3.1-2.0/cfg/CONFIG_PVACCESS_MODULE configure/CONFIG_SITE configure/RULES_TOP /cds/group/pcds/epics/base/R7.0.3.1-2.0/configure/RULES_TOP /cds/group/pcds/epics/base/R7.0.3.1-2.0/configure/RULES_DIRS /cds/group/pcds/epics/base/R7.0.3.1-2.0/configure/RULES_COMMON",


    "UNINSTALL_DIRS": "./archive ./autosave ./cfg ./db ./dbd ./doc ./html ./include  ./templates ./adl ./alh ./cfg ./edl ./lib/perl ./lib/pkgconfig ./restore ./db ./db",

Motivation and Context

Introspecting parts of the EPICS build system
Will be used in other tools for auto-downloading/building dependencies
Will meet up with ENH: docker container test ioc-machine-core#4 at some point

How Has This Been Tested?

Included test suite

Where Has This Been Documented?

This PR text

klauer · 2022-03-31T22:24:51Z

I think this is good enough as a first pass. Test coverage is 90%-ish for this PR.

➡️ Requesting at least a smell check, leaving level of review up to reviewer's whim

klauer · 2022-03-31T22:29:27Z

whatrecord/bin/parse.py

@@ -104,132 +96,6 @@ def build_arg_parser(parser=None):
    return parser


-ParseResult = Union[


This big block got moved to whatrecord.parse; and Makefile was added to the supported list

klauer · 2022-04-01T16:52:02Z

OK, ~~pending tests passing~~, this is really ready this time.
And perhaps this better represents the insanity that is EPICS module dependencies than the above simple example:

ads-ioc R0.5.0:

ZLLentz · 2022-04-04T16:07:59Z

Wow that's a gnarly dependency tree

whatrecord/bin/deps.py

tangkong

I'm not experienced enough with EPICS to be a great judge of this functionality, but it looks useful and well put-together to me. Tests look good too.

I should really play around with this more to get a better grasp on things, but at this moment it LGTM

ZLLentz · 2022-04-04T17:44:38Z

whatrecord/makefile.py

+
+    @classmethod
+    def from_makefile(
+        cls,


nitpick: this is a @classmethod that doesn't use the cls variable

Not a nitpick - a bug.

Edit: resolved in 5102519

ZLLentz

I didn't fully grasp how this works at the core (lack of Makefile knowledge), but the result is good and I only had minor nitpicks

klauer added 6 commits March 29, 2022 16:59

WIP: add Makefile information helper

44d5274

MNT: make special vars; conform to whatrecord's from_* naming

787397f

TST: makefile default goal

a36b167

MNT: make IocMetadata conform to naming too

deac2ff

MNT: rearrange parse tools to be more accessible

64fa317

ENH: add Makefile to general parser tool

f350361

klauer changed the title ~~WIP: add Makefile information helper~~ ENH: add Makefile information helper Mar 30, 2022

klauer added 6 commits March 30, 2022 09:50

TST: fix up parse import

945231f

MNT: add FileFormat and parse to top-level

8273077

MNT: add working directory and parse more variables

b41ad40

TST: add test for additional makefile info

cad321c

TST: correct test ID

3501c23

TST: show make version in testing and fail for now

21f6dbb

This comment was marked as outdated.

Sign in to view

MNT: add release_tops

92f6606

klauer force-pushed the makefile_helper branch from 14b08ac to 2a42ede Compare March 31, 2022 00:02

MNT: echo -e and null terminators, not so portable

7515ba3

klauer force-pushed the makefile_helper branch from 2a42ede to 7515ba3 Compare March 31, 2022 16:37

klauer added 2 commits March 31, 2022 12:24

WIP: 'whatrecord deps' for graphing/inspecting IOC deps

8ccfcc7

MNT: support 'whatrecord graph Makefile'

f9864f7

This comment was marked as outdated.

Sign in to view

klauer marked this pull request as ready for review March 31, 2022 22:22

klauer requested review from tangkong and ZLLentz March 31, 2022 22:25

klauer commented Mar 31, 2022

View reviewed changes

klauer added 3 commits March 31, 2022 15:38

DOC: add whatrecord graph to readme

1fdc9c1

FIX: fix and test dependency graphs

e9729dd

MNT: remove silent failure option + better coverage

e7783b9

ZLLentz reviewed Apr 4, 2022

View reviewed changes

whatrecord/bin/deps.py Outdated Show resolved Hide resolved

MNT: fix whatrecord deps docstring

022165a

tangkong approved these changes Apr 4, 2022

View reviewed changes

ZLLentz reviewed Apr 4, 2022

View reviewed changes

ZLLentz approved these changes Apr 4, 2022

View reviewed changes

MNT: use cls.from_makefile in the classmethod

5102519

klauer merged commit 97b22a6 into pcdshub:master Apr 4, 2022

klauer deleted the makefile_helper branch April 4, 2022 18:16

klauer mentioned this pull request Apr 4, 2022

Add dependency version information #84

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ENH: add Makefile information helper #124

ENH: add Makefile information helper #124

klauer commented Mar 30, 2022 •

edited

Loading

This comment was marked as outdated.

This comment was marked as outdated.

klauer commented Mar 31, 2022 •

edited

Loading

klauer Mar 31, 2022

klauer commented Apr 1, 2022 •

edited

Loading

ZLLentz commented Apr 4, 2022

tangkong left a comment

ZLLentz Apr 4, 2022

klauer Apr 4, 2022 •

edited

Loading

ZLLentz left a comment

		@@ -104,132 +96,6 @@ def build_arg_parser(parser=None):
		return parser


		ParseResult = Union[

ENH: add Makefile information helper #124

ENH: add Makefile information helper #124

Conversation

klauer commented Mar 30, 2022 • edited Loading

Description

Motivation and Context

How Has This Been Tested?

Where Has This Been Documented?

This comment was marked as outdated.

This comment was marked as outdated.

klauer commented Mar 31, 2022 • edited Loading

klauer Mar 31, 2022

Choose a reason for hiding this comment

klauer commented Apr 1, 2022 • edited Loading

ZLLentz commented Apr 4, 2022

tangkong left a comment

Choose a reason for hiding this comment

ZLLentz Apr 4, 2022

Choose a reason for hiding this comment

klauer Apr 4, 2022 • edited Loading

Choose a reason for hiding this comment

ZLLentz left a comment

Choose a reason for hiding this comment

klauer commented Mar 30, 2022 •

edited

Loading

klauer commented Mar 31, 2022 •

edited

Loading

klauer commented Apr 1, 2022 •

edited

Loading

klauer Apr 4, 2022 •

edited

Loading