zoukankan      html  css  js  c++  java
  • 12、pytest -- 缓存:记录执行的状态

    往期索引:https://www.cnblogs.com/luizyao/p/11771740.html

    pytest会将本轮测试的执行状态写入到.pytest_cache文件夹,这个行为是由自带的cacheprovider插件来实现的;

    注意:

    pytest默认将测试执行的状态写入到根目录中的.pytest_cache文件夹,我们也可以通过在pytest.ini中配置cache_dir选项来自定义缓存的目录,它可以是相对路径,也可以是绝对路径;

    相对路径指的是相对于pytest.ini文件所在的目录;例如,我们把这一章的缓存和源码放在一起:

    src/chapter-12/pytest.ini中添加如下配置:

    [pytest]
    cache_dir = .pytest-cache
    

    这样,即使我们在项目的根目录下执行src/chapter-12/中的用例,也只会在pytest-chinese-doc/src/chapter-12/.pytest_cache中生成缓存,而不再是pytest-chinese-doc/.pytest_cache中;

    pytest-chinese-doc (5.1.3) 
    λ pipenv run pytest src/chapter-12
    

    1. cacheprovider插件

    在介绍这个插件之前,我们先看一个简单例子:

    # src/chapter-12/test_failed.py
    
    import pytest
    
    
    @pytest.mark.parametrize('num', [1, 2])
    def test_failed(num):
        assert num == 1
    
    
    # srcchapter-12	est_pass.py
    
    def test_pass():
        assert 1
    

    我们有两个简单的测试模块,首先我们来执行一下它们:

    λ pipenv run pytest -q src/chapter-12/
    .F.                                                                [100%] 
    =============================== FAILURES ================================ 
    ____________________________ test_failed[2] _____________________________
    
    num = 2
    
        @pytest.mark.parametrize('num', [1, 2])
        def test_failed(num):
    >       assert num == 1
    E       assert 2 == 1
    
    srcchapter-12	est_failed.py:27: AssertionError
    1 failed, 2 passed in 0.08s
    

    可以看到一共收集到三个测试用例,其中有一个失败,另外两个成功的,并且两个执行成功的用例分属不同的测试模块;

    同时,pytest也在src/chapter-12/的目录下生成缓存文件夹(.pytest_cache),具体的目录结构如下所示:

    src
    ├───chapter-12
    │   │   pytest.ini  # 配置了 cache_dir = .pytest-cache
    │   │   test_failed.py
    │   │   test_pass.py
    │   │
    │   └───.pytest-cache
    │       │   .gitignore
    │       │   CACHEDIR.TAG
    │       │   README.md
    │       │
    │       └───v
    │           └───cache
    │                   lastfailed
    │                   nodeids
    │                   stepwise
    

    现在,我们就结合上面的组织结构,具体介绍一下cacheprovider插件的功能;

    1.1. --lf, --last-failed:只执行上一轮失败的用例

    缓存中的lastfailed文件记录了上次失败的用例ID,我们可以通过一下--cache-show命令查看它的内容:

    --cache-show命令也是cacheprovider提供的新功能,它不会导致任何用例的执行;

    λ pipenv run pytest src/chapter-12/ -q --cache-show 'lastfailed'
    cachedir: D:Personal FilesProjectspytest-chinese-docsrcchapter-12.pytest-cache
    --------------------- cache values for 'lastfailed' --------------------- 
    cachelastfailed contains:
      {'test_failed.py::test_failed[2]': True}
    
    no tests ran in 0.01s
    

    我们可以看到,它记录了一个用例,为上次失败的测试用例的IDtest_failed.py::test_failed[2]

    下次执行时,当我们使用--lf选项,pytest在收集阶段只会选择这个失败的用例,而忽略其它的:

    λ pipenv run pytest --lf --collect-only src/chapter-12/
    ========================== test session starts ==========================
    platform win32 -- Python 3.7.3, pytest-5.1.3, py-1.8.0, pluggy-0.13.0
    cachedir: .pytest-cache
    rootdir: D:Personal FilesProjectspytest-chinese-docsrcchapter-12, inifile: pytest.ini
    collected 2 items / 1 deselected / 1 selected
    <Module test_failed.py>
      <Function test_failed[2]>
    run-last-failure: rerun previous 1 failure (skipped 2 files)
    
    ========================= 1 deselected in 0.02s =========================
    

    我们仔细观察一下上面的回显,有一句话可能会让我们有点困惑:collected 2 items / 1 deselected / 1 selected,可我们明明有三个用例,怎么会只收集到两个呢?

    实际上,--lf复写了用例收集阶段的两个钩子方法:pytest_ignore_collect(path, config)pytest_collection_modifyitems(session, config, items)

    我们来先看看pytest_ignore_collect(path, config),如果它的结果返回True,就忽略path路径中的用例;

    # _pytest/cacheprovider.py
    
        def last_failed_paths(self):
            """Returns a set with all Paths()s of the previously failed nodeids (cached).
            """
            try:
                return self._last_failed_paths
            except AttributeError:
                rootpath = Path(self.config.rootdir)
                result = {rootpath / nodeid.split("::")[0] for nodeid in self.lastfailed}
                result = {x for x in result if x.exists()}
                self._last_failed_paths = result
                return result
    
        def pytest_ignore_collect(self, path):
            """
            Ignore this file path if we are in --lf mode and it is not in the list of
            previously failed files.
            """
            if self.active and self.config.getoption("lf") and path.isfile():
                last_failed_paths = self.last_failed_paths()
                if last_failed_paths:
                    skip_it = Path(path) not in self.last_failed_paths()
                    if skip_it:
                        self._skipped_files += 1
                    return skip_it
    

    可以看到,如果当前收集的文件,不在上一次失败的路径集合内,就会忽略这个文件,所以这次执行就不会到test_pass.py中收集用例了,故而只收集到两个用例;并且pytest.ini也在忽略的名单上,所以实际上是跳过两个文件:(skipped 2 files)

    至于pytest_collection_modifyitems(session, config, items)钩子方法,我们在下一节和--ff命令一起看;

    1.2. --ff, --failed-first:先执行上一轮失败的用例,再执行其它的

    我们先通过实践看看这个命令的效果,再去分析它的实现:

    λ pipenv run pytest --collect-only -s --ff src/chapter-12/
    ========================== test session starts ========================== 
    platform win32 -- Python 3.7.3, pytest-5.1.3, py-1.8.0, pluggy-0.13.0
    cachedir: .pytest-cache
    rootdir: D:Personal FilesProjectspytest-chinese-docsrcchapter-12, inifile: pytest.ini
    collected 3 items
    <Module test_failed.py>
      <Function test_failed[2]>
      <Function test_failed[1]>
    <Module test_pass.py>
      <Function test_pass>
    run-last-failure: rerun previous 1 failure first
    
    ========================= no tests ran in 0.02s =========================
    

    我们可以看到一共收集到三个测试用例,和正常的收集顺序相比,上一轮失败的test_failed.py::test_failed[2]用例在最前面,将优先执行;

    实际上,-ff只复写了钩子方法:pytest_collection_modifyitems(session, config, items),它可以过滤或者重新排序收集到的用例:

    # _pytest/cacheprovider.py
    
        def pytest_collection_modifyitems(self, session, config, items):
            ...
    
                    if self.config.getoption("lf"):
                        items[:] = previously_failed
                        config.hook.pytest_deselected(items=previously_passed)
                    else:  # --failedfirst
                        items[:] = previously_failed + previously_passed
    
            ...
    

    可以看到,如果使用的是lf,就把之前成功的用例状态置为deselected,这轮执行就会忽略它们;如果使用的是-ff,只是将之前失败的用例,顺序调到前面;

    另外,我们也可以看到lf的优先级要高于ff,所以它们同时使用的话,ff是不起作用的;

    1.3. --nf, --new-first:先执行新加的或修改的用例,再执行其它的

    缓存中的nodeids文件记录了上一轮执行的所有的用例:

    λ pipenv run pytest src/chapter-12 --cache-show 'nodeids'
    ========================== test session starts ==========================
    platform win32 -- Python 3.7.3, pytest-5.1.3, py-1.8.0, pluggy-0.13.0
    cachedir: .pytest-cache
    rootdir: D:Personal FilesProjectspytest-chinese-docsrcchapter-12, inifile: pytest.ini
    cachedir: D:Personal FilesProjectspytest-chinese-docsrcchapter-12.pytest-cache
    ---------------------- cache values for 'nodeids' -----------------------
    cache
    odeids contains:
      ['test_failed.py::test_failed[1]',
       'test_failed.py::test_failed[2]',
       'test_pass.py::test_pass']
    
    ========================= no tests ran in 0.01s =========================
    

    我们看到上一轮共执行了三个测试用例;

    现在我们在test_pass.py中新加一个用例,并修改一下test_failed.py文件中的用例(但是不添加新用例):

    # srcchapter-12	est_pass.py
    
    def test_pass():
        assert 1
    
    
    def test_new_pass():
        assert 1
    

    现在我们再来执行一下收集命令:

    λ pipenv run pytest --collect-only -s --nf src/chapter-12/
    ========================== test session starts ==========================
    platform win32 -- Python 3.7.3, pytest-5.1.3, py-1.8.0, pluggy-0.13.0
    cachedir: .pytest-cache
    rootdir: D:Personal FilesProjectspytest-chinese-docsrcchapter-12, inifile: pytest.ini
    collected 4 items
    <Module test_pass.py>
      <Function test_new_pass>
    <Module test_failed.py>
      <Function test_failed[1]>
      <Function test_failed[2]>
    <Module test_pass.py>
      <Function test_pass>
    
    ========================= no tests ran in 0.03s =========================
    

    可以看到,新加的用例顺序在最前面,其次修改过的测试用例紧接其后,最后才是旧的用例;这个行为在源码中有所体现:

    # _pytest/cacheprovider.py
    
        def pytest_collection_modifyitems(self, session, config, items):
            if self.active:
                new_items = OrderedDict()
                other_items = OrderedDict()
                for item in items:
                    if item.nodeid not in self.cached_nodeids:
                        new_items[item.nodeid] = item
                    else:
                        other_items[item.nodeid] = item
    
                items[:] = self._get_increasing_order(
                    new_items.values()
                ) + self._get_increasing_order(other_items.values())
            self.cached_nodeids = [x.nodeid for x in items if isinstance(x, pytest.Item)]
    
        def _get_increasing_order(self, items):
            return sorted(items, key=lambda item: item.fspath.mtime(), reverse=True)
    

    item.fspath.mtime()代表用例所在文件的最后修改时间,reverse=True表明是倒序排列;

    items[:] = self._get_increasing_order(new_items.values()) + self._get_increasing_order(other_items.values())保证新加的用例永远在最前面;

    1.4. --cache-clear:先清除所有缓存,再执行用例

    直接看源码:

    # _pytest/cacheprovider.py
    
    class Cache:
    
        ... 
    
        @classmethod
        def for_config(cls, config):
            cachedir = cls.cache_dir_from_config(config)
            if config.getoption("cacheclear") and cachedir.exists():
                rm_rf(cachedir)
                cachedir.mkdir()
            return cls(cachedir, config)
    

    可以看到,它会先把已有的缓存文件夹删除(rm_rf(cachedir)),再创建一个空的同名文件夹(cachedir.mkdir()),这样会导致上述的功能失效,所以一般不使用这个命令;

    1.5. 如果上一轮没有失败的用例

    现在,我们清除缓存,再执行test_pass.py模块(它的用例都是能测试成功的):

    λ pipenv run pytest --cache-clear -q -s src/chapter-12/test_pass.py
    .
    1 passed in 0.01s
    

    这时候我们再去看一下缓存目录:

    .pytest-cache
    └───v
        └───cache
                nodeids
                stepwise
    

    是不是少了什么?对!因为没有失败的用例,所以不会生成lastfailed文件,那么这个时候在使用--lf--ff会发生什么呢?我们来试试:

    注意:

    如果我们观察的足够仔细,就会发现现在的缓存目录和之前相比不止少了lastfailed文件,还少了CACHEDIR.TAG.gitignoreREADME.md三个文件;

    这是一个bug,我已经在pytest 5.3.1版本上提交了issue,预计会在之后的版本修复,如果你有兴趣深入了解一下它的成因和修复方案,可以参考这个:https://github.com/pytest-dev/pytest/issues/6290

    luyao@NJ-LUYAO-T460 /d/Personal Files/Projects/pytest-chinese-doc (5.1.3) 
    λ pipenv run pytest -q -s --lf src/chapter-12/test_pass.py
    .
    1 passed in 0.01s
    
    luyao@NJ-LUYAO-T460 /d/Personal Files/Projects/pytest-chinese-doc (5.1.3) 
    λ pipenv run pytest -q -s --ff src/chapter-12/test_pass.py
    .
    1 passed in 0.02s
    

    可以看到,它们没有实施任何影响;为什么会这样?我们去源码里找一下答案吧;

    # _pytest/cacheprovider.py
    
    class LFPlugin:
        """ Plugin which implements the --lf (run last-failing) option """
    
        def __init__(self, config):
            ...
            self.lastfailed = config.cache.get("cache/lastfailed", {})
            ...
    
        def pytest_collection_modifyitems(self, session, config, items):
            ...
    
            if self.lastfailed:
    
                ...
    
            else:
                self._report_status = "no previously failed tests, "
                if self.config.getoption("last_failed_no_failures") == "none":
                    self._report_status += "deselecting all items."
                    config.hook.pytest_deselected(items=items)
                    items[:] = []
                else:
                    self._report_status += "not deselecting items."
    

    可以看到,当self.lastfailed判断失败时,如果我们指定了last_failed_no_failures选项为nonepytest会忽略所有的用例(items[:] = []),否则不做任何修改(和没加--lf--ff一样),而判断self.lastfailed的依据是就是lastfailed文件;

    继续看看,我们会学习到一个新的命令行选项:

    # _pytest/cacheprovider.py
    
        group.addoption(
                "--lfnf",
                "--last-failed-no-failures",
                action="store",
                dest="last_failed_no_failures",
                choices=("all", "none"),
                default="all",
                help="which tests to run with no previously (known) failures.",
            )
    

    来试试吧:

    λ pipenv run pytest -q -s --ff --lfnf none src/chapter-12/test_pass.py
    
    1 deselected in 0.01s
    
    λ pipenv run pytest -q -s --ff --lfnf all src/chapter-12/test_pass.py
    .
    1 passed in 0.01s
    

    注意:

    --lfnf的实参只支持choices=("all", "none")

    2. config.cache对象

    我们可以通过pytestconfig对象去访问和设置缓存中的数据;下面是一个简单的例子:

    # content of test_caching.py
    
    import pytest
    import time
    
    
    def expensive_computation():
        print("running expensive computation...")
    
    
    @pytest.fixture
    def mydata(request):
        val = request.config.cache.get("example/value", None)
        if val is None:
            expensive_computation()
            val = 42
            request.config.cache.set("example/value", val)
        return val
    
    
    def test_function(mydata):
        assert mydata == 23
    

    我们先执行一次这个测试用例:

    λ pipenv run pytest -q src/chapter-12/test_caching.py 
    F                                                                   [100%]
    ================================ FAILURES =================================
    ______________________________ test_function ______________________________
    
    mydata = 42
    
        def test_function(mydata):
    >       assert mydata == 23
    E       assert 42 == 23
    
    src/chapter-12/test_caching.py:43: AssertionError
    -------------------------- Captured stdout setup --------------------------
    running expensive computation...
    1 failed in 0.05s
    

    这个时候,缓存中没有example/value,将val的值写入缓存,终端打印running expensive computation...

    查看缓存,其中新加了一个文件:.pytest-cache/v/example/value

    .pytest-cache/
    ├── .gitignore
    ├── CACHEDIR.TAG
    ├── README.md
    └── v
        ├── cache
        │   ├── lastfailed
        │   ├── nodeids
        │   └── stepwise
        └── example
            └── value
    
    3 directories, 7 files
    

    通过--cache-show选项查看,发现其内容正是42

    λ pipenv run pytest src/chapter-12/ -q --cache-show 'example/value'
    cachedir: /Users/yaomeng/Private/Projects/pytest-chinese-doc/src/chapter-12/.pytest-cache
    -------------------- cache values for 'example/value' ---------------------
    example/value contains:
      42
    
    no tests ran in 0.00s
    

    再次执行这个用例,这个时候缓存中已经有我们需要的数据了,终端就不会再打印running expensive computation...

    λ pipenv run pytest -q src/chapter-12/test_caching.py 
    F                                                                   [100%]
    ================================ FAILURES =================================
    ______________________________ test_function ______________________________
    
    mydata = 42
    
        def test_function(mydata):
    >       assert mydata == 23
    E       assert 42 == 23
    
    src/chapter-12/test_caching.py:43: AssertionError
    1 failed in 0.04s
    

    3. Stepwise

    试想一下,现在有这么一个场景:我们想要在遇到第一个失败的用例时退出执行,并且下次还是从这个用例开始执行;

    以下面这个测试模块为例:

    # src/chapter-12/test_sample.py
    
    def test_one():
        assert 1
    
    
    def test_two():
        assert 0
    
    
    def test_three():
        assert 1
    
    
    def test_four():
        assert 0
    
    
    def test_five():
        assert 1
    

    我们先执行一下测试:pipenv run pytest --cache-clear --sw src/chapter-12/test_sample.py

    λ pipenv run pytest --cache-clear --sw -q src/chapter-12/test_sample.py
    .F
    ================================= FAILURES =================================
    _________________________________ test_two _________________________________
    
        def test_two():
    >       assert 0
    E       assert 0
    
    src/chapter-12/test_sample.py:28: AssertionError
    !!!!!! Interrupted: Test failed, continuing from this test next run. !!!!!!!
    1 failed, 1 passed in 0.13s
    

    使用--cache-clear清除之前的缓存,使用--sw, --stepwise使其在第一个失败的用例处退出执行;

    现在我们的缓存文件中lastfailed记录了这次执行失败的用例,即为test_two()nodeids记录了所有的测试用例;特殊的是,stepwise记录了最近一次失败的测试用例,这里也是test_two()

    接下来,我们用--sw的方式再次执行:pytest首先会读取stepwise中的值,并将其作为第一个用例开始执行;

    λ pipenv run pytest --sw -q src/chapter-12/test_sample.py
    F
    ================================= FAILURES =================================
    _________________________________ test_two _________________________________
    
        def test_two():
    >       assert 0
    E       assert 0
    
    src/chapter-12/test_sample.py:28: AssertionError
    !!!!!! Interrupted: Test failed, continuing from this test next run. !!!!!!!
    1 failed, 1 deselected in 0.12s
    

    可以看到,test_two()作为第一个用例开始执行,在第一个失败处退出;

    其实,pytest还提供了一个--stepwise-skip的命令行选项,它会忽略第一个失败的用例,在第二个失败处退出执行;我们来试一下:

    λ pipenv run pytest --sw --stepwise-skip -q src/chapter-12/test_sample.py
    F.F
    =============================== FAILURES ================================ 
    _______________________________ test_two ________________________________
    
        def test_two():
    >       assert 0
    E       assert 0
    
    srcchapter-12	est_sample.py:28: AssertionError
    _______________________________ test_four _______________________________
    
        def test_four():
    >       assert 0
    E       assert 0
    
    srcchapter-12	est_sample.py:36: AssertionError
    !!!!! Interrupted: Test failed, continuing from this test next run. !!!!! 2 failed, 1 passed, 1 deselected in 0.16s
    

    这个时候,在第二个失败的用例test_four()处退出执行,同时stepwise文件的值也改成了"test_sample.py::test_four"

    其实,本章所有的内容都可以在源码的_pytest/cacheprovider.py文件中体现,如果能结合源码学习,会有事半功倍的效果;

  • 相关阅读:
    xls与csv文件的区别
    青音,经典爱情语录
    win7用户账户自动登录方法汇总
    How to using Procedure found Lead Blocker
    FTS(3) BSD 库函数手册 遍历文件夹(二)
    FTS(3) BSD 库函数手册 遍历文件夹(一)
    DisplayMetrics类 获取手机显示屏的基本信息 包括尺寸、密度、字体缩放等信息
    About App Distribution 关于应用发布
    FTS(3) 遍历文件夹实例
    OpenCV 2.1.0 with Visual Studio 2008
  • 原文地址:https://www.cnblogs.com/luizyao/p/11969452.html
Copyright © 2011-2022 走看看