Skip to content

Add figleaf coverage metrics #46655

Closed
Closed
Add figleaf coverage metrics#46655
Labels
testsTests in the Lib/test dirtype-featureA feature request or enhancement

Description

@jerryseutter
jerryseutter
mannequin
opened on Mar 18, 2008
BPO 2403
Nosy @ncoghlan, @giampaolo, @devdanzin, @merwok, @sandrotosi
Files
  • README.patch: Update Tools/README
  • coverage.zip: Tools/coverage directory containing files
  • coverage.zip: Tools/coverage .zip revision 2
  • coverage.py: Main file in coverage.zip
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = <Date 2014-07-06.06:54:21.417>
created_at = <Date 2008-03-18.19:42:54.725>
labels = ['type-feature', 'tests']
title = 'Add figleaf coverage metrics'
updated_at = <Date 2014-07-06.06:54:21.416>
user = 'https://bugs.python.org/jerryseutter'

    bugs.python.org fields:

    activity = <Date 2014-07-06.06:54:21.416>
actor = 'berker.peksag'
assignee = 'none'
closed = True
closed_date = <Date 2014-07-06.06:54:21.417>
closer = 'berker.peksag'
components = ['Tests']
creation = <Date 2008-03-18.19:42:54.725>
creator = 'jerry.seutter'
dependencies = []
files = ['9737', '9738', '9744', '13788']
hgrepos = []
issue_num = 2403
keywords = ['patch']
message_count = 14.0
messages = ['63975', '63976', '63979', '64005', '64008', '64025', '64053', '86580', '112228', '112229', '154941', '154987', '213358', '222383']
nosy_count = 8.0
nosy_names = ['nnorwitz', 'ncoghlan', 'jerry.seutter', 'giampaolo.rodola', 'ajaksu2', 'eric.araujo', 'sandro.tosi', 'BreamoreBoy']
pr_nums = []
priority = 'low'
resolution = 'rejected'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue2403'
versions = ['Python 3.5']

    Activity

    1. jerryseutter commented on Mar 18, 2008

      @jerryseutter
      jerryseuttermannequin
      on Mar 18, 2008
      MannequinAuthor

      This issue adds support for figleaf unit test coverage information. The
      diffs apply against trunk

    2. jerryseutter
      added
      testsTests in the Lib/test dir
      on Mar 18, 2008

    3. jerryseutter commented on Mar 18, 2008

      @jerryseutter
      jerryseuttermannequin
      on Mar 18, 2008
      MannequinAuthor

      To test:

      1. Unzip the zipfile in the base python directory. The zipfile will
        create Tools/coverage*.
      2. cd Tools; patch -p0 README.patch
      3. cd coverage
      4. ../../python.exe coverage.py

      The script will download figleaf, then run regrtest.py. Any extra stuff
      on the command line will be supplied to regrtest.

    4. nnorwitz commented on Mar 18, 2008

      @nnorwitz
      nnorwitzmannequin
      on Mar 18, 2008
      Mannequin

      Thanks for the patch. It would be nice to get more instrumentation like
      coverage, performance, etc.

      Here are some things I noticed while reviewing the patch:

      • This won't work on unix other than OSX. Can you change
        ../../python.exe to sys.executable?
      • How big is figleaf? Should you try to read/write the file in chunks?
      • optparse doesn't seem to be used.
      • Can you change cat file to read the file and pass those as
        arguments so this could work on windows?

      BTW, does figleaf work on Windows?

    5. jerryseutter commented on Mar 18, 2008

      @jerryseutter
      jerryseuttermannequin
      on Mar 18, 2008
      MannequinAuthor

      Thanks for the input.

      • ../../python.exe changed to sys.executable.
      • Figleaf is 69kb. It seems to work fine doing it all in one read()
        call. Should it be chunked?
      • Optparse isn't used, partially because I'm lazy and partly because
        I'm not actually parsing the command line, just chopping the first
        element off and then passing it off to regrtest.py. It is the job of
        regrtest.py to parse the command line. <-- My opinion
      • traceless. I think I should remove the file. Having tests that fail
        don't hurt figleaf, as far as I know, so there isn't much point to
        avoiding failing tests.

      Uploading a new version of the .zip file that uses sys.executable and
      does not use traceless.

      I don't know if figleaf works on windows and I don't have a system to
      test with.

    6. nnorwitz commented on Mar 18, 2008

      @nnorwitz
      nnorwitzmannequin
      on Mar 18, 2008
      Mannequin
      • ../../python.exe changed to sys.executable.
      • Figleaf is 69kb. It seems to work fine doing it all in one read()
        call. Should it be chunked?

      At 69kb, nah. It should be good enough for the first cut.

      • Optparse isn't used, partially because I'm lazy and partly because
        I'm not actually parsing the command line, just chopping the first
        element off and then passing it off to regrtest.py. It is the job of
        regrtest.py to parse the command line. <-- My opinion

      Oh sorry, I only meant to remove the import since it wasn't used. I
      wasn't secretly trying to get you to do more work (at least not yet).
      :-)

    7. smontanaro commented on Mar 19, 2008

      @smontanaro
      smontanaro
      on Mar 19, 2008
      Contributor

      How will this work if I use a build directory? For example,
      my source is in ~/src/python/trunk. In there I create a
      build directory. When I run it I get

      % pwd
/Users/skip/src/python/trunk/build
% ./python.exe ../Tools/coverage/coverage.py 
Running tests...
Traceback (most recent call last):
  File "figleaf-latest/bin/figleaf", line 4, in <module>
    figleaf.main()
  File
"/Users/skip/src/python/trunk/build/figleaf-latest/figleaf/__init__.py",
line 302, in main
    execfile(sys.argv[0], __main__.__dict__)
IOError: [Errno 2] No such file or directory: '../../Lib/test/regrtest.py'
Generating html...
CANNOT OPEN: @test
figleaf: HTML output written to ../../coverage

      I think this mode should work. I prefer not to pollute my
      source tree with build information.

      Skip

    8. smontanaro commented on Mar 19, 2008

      @smontanaro
      smontanaro
      on Mar 19, 2008
      Contributor

      I gave this a try. It seems to not report on many files. For example,
      test_csv was run and passed, but there is no html file in the coverage
      directory with "csv" in its name after figleaf2html is run. Nor is there a
      key in the pickled dictionary in the .figleaf file which includes the string
      "csv".

      Skip

    9. devdanzin commented on Apr 26, 2009

      @devdanzin
      devdanzinmannequin
      on Apr 26, 2009
      Mannequin

      Re-attaching the coverage bootstrapping file as plain/text.

    10. devdanzin
      added
      type-featureA feature request or enhancement
      on Apr 26, 2009

    11. merwok commented on Jul 31, 2010

      @merwok
      merwok
      on Jul 31, 2010
      Member

      IIUC, using runpy (either from Python or in a subprocess i.e. python -m test.regrtest) would allow running from build dir.

      Coverage reports are nice, but I personally dislike the idea of code downloading something via HTTP and running it. I’d prefer just having an optional dependency on it and letting the user install it if they want.

      Code review:

      1. There is already a popular (I think) module named coverage, please use another name.
        1.1) How does figleaf compare with coverage?
      2. Please use tarfile instead of running tar (not always available).
      3. Please use the Python API instead of running subprocesses.
      4. subprocess is recommended over os.system. Neither will be required if you agree to 2 and 3 :)
      5. Please make the code PEP-8-compliant.

      Hope this helps! :)

    12. merwok commented on Jul 31, 2010

      @merwok
      merwok
      on Jul 31, 2010
      Member

      the Python API of figleaf*

    13. merwok commented on Mar 5, 2012

      @merwok
      merwok
      on Mar 5, 2012
      Member

      Adding people interested in code coverage to nosy.

    14. sandrotosi commented on Mar 5, 2012

      @sandrotosi
      sandrotosi
      on Mar 5, 2012
      Contributor

      The devguide reports a long section about code convering cpython: http://docs.python.org/devguide/coverage.html . It uses Ned's coverege.py and has a lot of details about how to use it and how to generate nice reports out of it. do we need an internal tool for cpython coverage?

    15. merwok commented on Mar 13, 2014

      @merwok
      merwok
      on Mar 13, 2014
      Member

      Indeed, the devguide documents how to use coverage.py, and there is even a pure-stdlib solution with regrtest. I recommend closing this.

    16. BreamoreBoy commented on Jul 6, 2014

      @BreamoreBoy
      BreamoreBoymannequin
      on Jul 6, 2014
      Mannequin

      As there have been no replies to msg213358 can someone close this please.

    17. berkerpeksag
      closed this as completedon Jul 6, 2014
    18. ezio-melotti
      transferred this issue fromon Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    No one assigned

      Labels

      testsTests in the Lib/test dirtype-featureA feature request or enhancement

      Projects

      No projects

        Milestone

        No milestone

        Relationships

        None yet

        Development

        No branches or pull requests

        Issue actions