Unit test updates by slide · Pull Request #2 · IronLanguages/ironpython2

slide · 2017-05-25T23:30:03Z

Updates to IronPython specific unit tests to use the unittest

slozier

Wow, good work. I'm going to have to go through this a few times.

Are the commented out tests things you still need to work on?

slozier · 2017-05-26T13:23:44Z

-    AreEqual(0, launch_ironpython_changing_extensions(__file__, remove=["-O"]))
-else:
-    run_test(__name__)
+# if is_cli and '-O' in System.Environment.GetCommandLineArgs():


Could we retain this check in some form and make the test fail (or be skipped) if optimize is set? Maybe something like

@unittest.skipIf(sys.flags.optimize, "should be run without optimize")

would work?

It doesn't quite do what was originally there, but we could add the check for optimized.

slozier · 2017-05-26T13:26:01Z

-    #my_name             = os.environ.get(r'USERNAME', None)
-    #my_dir              = my_name and path_combine(team_dir, my_name) or None
-    #my_profile          = my_dir and path_combine(my_dir, r'settings.py') or None
+#     #team_dir            = path_combine(ip_root, r'Team')


If this was already commented might as well just zap it.

No, it's part of what makes this WIP still...I need it to refer back to as I work through test. It will be removed in the final commit.

slozier · 2017-05-26T13:32:06Z

 #####################################################################################

-AreEqual(z, 10)
+self.assertEqual(z, 10)


I'm guessing more work needs to be done here?

No, this is imported in test_execfile.py, it's just used for testing.

slozier · 2017-05-26T13:37:41Z

-from iptest.assert_util import *
-
-Assert(__name__ in  ["__main__", "test_execfile"], __name__)
+self.assertTrue(__name__ in  ["__main__", "test_execfile"], __name__)


Still working on this one?

No, this is imported in test_execfile.py

slozier · 2017-05-26T13:40:28Z

+        a = x()
+        try:
+            hasattr(a, 'abc')
+            AssertUnreachable()


Replace with unittest equivalent?

Yes, when it works I don't see the errors for wrong assertion methods :-)

slozier · 2017-05-26T14:01:46Z

-    import clr
-cp946()
+    def test_no_clr_import(self):
+        self.cp946()


Does unittest guarantee that this one will run before the next one?

No, but isn't the import local to the method?

Yes the clr variable is local to the function, but just fact that we're importing clr causes some side effects (such as the C# methods becoming visible). For example:

def test(): import clr assert "ToString" not in dir("a") test() assert "ToString" in dir("a")

I'll rename the methods, there is this note in the unittest docs:

"Note that the order in which the various test cases will be run is determined by sorting the test function names with respect to the built-in ordering for strings."

They will run in the correct order now, but I'll add a 00 an 01 respectively just to make sure.

Maybe we could add something like assert not hasattr(object, "ToString") to check that clr isn't loaded?

slozier · 2017-05-26T14:07:06Z

+
+from iptest import IronPythonTestCase, is_cli
+
+@unittest.skipUnless(is_cli, 'IronPython specific test case')


The skip was only on test_sanity. Although unless there's something cli specific in toexec.py it might not be necessary.

You are right, I don't think it's needed at all.

slozier · 2017-05-26T14:10:31Z

 #  Copyright (c) Microsoft Corporation. All rights reserved.
 #
-# This source code is subject to terms and conditions of the Apache License, Version 2.0. A
+# This source code is subject to terms and conditions of the Apache License, Version 2.0. self.A


Added a typo?

slozier · 2017-05-26T14:12:10Z

@@ -1,142 +0,0 @@
-#####################################################################################


Where did this file one go?

No tests were being run from this file and the framework for running the test no longer exists in the testing libraries.

slozier · 2017-05-26T14:20:17Z

-    class D(C1): pass
-    d = D()
-    AreEqual([d.x, d.y], [100, 100])
+    def test_delete_getattribute(self):


This one was disabled. Is it fixed now?

Yes, it runs correctly. I generally have been enabling all the tests and then looking through to see if there really is something that should be disabled. I am also trying to use expectedFailure instead of disabling if there is an issue, that way it is noted in the logs.

slozier · 2017-05-26T19:54:56Z

        try:
            hasattr(a, 'abc')
-            AssertUnreachable()
+            self.fail('Should not reach this point')


I think there is an assertUnreachable in IronPythonTestCase? There are probably more AssertUnreachable all over the place...

slozier · 2017-05-26T19:55:04Z

-    import clr
-cp946()
+    def test_no_clr_import(self):
+        self.cp946()


Maybe we could add something like assert not hasattr(object, "ToString") to check that clr isn't loaded?

slozier · 2017-05-29T00:54:00Z

+            Fail("Returned value of getdebug method is not valid value:" + str(result))
+
+
+#CodePlex Work Item# 8202


I believe this comment should go with test_000_get_debug.

slozier · 2017-05-29T01:00:06Z

+        self.assertEqual(errno.errorcode[2],
+                "ENOENT")
+
+    @unittest.skipIf(is_posix, 'Test does not work on Mono')


Mono or posix? Do we have an is_mono?

slozier · 2017-05-29T01:36:35Z

-        return "MyException is a user sub-type of IOError"
-
-@skip("win32")
-def test_clr_exception_has_non_trivial_exception_message():


Did you move this one to another file?

slozier · 2017-05-29T01:42:27Z

+        l[-1] = p2
+        self.assertEqual(id(l[-1]), id(p2))
+
+    @unittest.skipIf(is_netstandard or not is_cli, 'netstandard does not support PlatformNotSupportedException, nor does native')


The skipIf message should probably read something like: IronPython specific test, .NET Core throws a PlatformNotSupportedException

slozier · 2017-05-29T01:43:48Z

+        self.assertTrue(is_complete)
+        self.assertEqual(Flag.Value, 32)
+
+    @unittest.skipIf(is_posix, "mono's GC behaves differently so we get another print when GC occurs")


in_mono instead of is_posix?

slozier · 2017-05-29T13:03:23Z

-def test_object():
-    a = object()
-    b = object()
+@unittest.skipUnless(is_cli, 'IronPython specific test case')


Move the skipUnless over to test_bool_nullablebool? test_object should work with CPython.

slozier · 2017-05-29T13:07:46Z

-except NameError:
-    pass
-
-def test_file():


Dropped this test?

Yes, it didn't seem to fit with the test of the tests.

slozier · 2017-05-29T13:16:13Z

+        self.assertRaises(NameError, lambda: [(i, j) for i in range(10) if j < 'c' for j in ['a', 'b', 'c'] if i % 3 == 0])
+
+    def test_negative_flow_checker(self):
+        self.assertRaises(NameError, lambda: [y for x in "python"])


This seems like exactly the same as the test above?

Yeah, I'll remove the second one.

slozier · 2017-05-29T13:37:19Z

+
+    def test_stdin(self):
+        self.assertTrue('<stdin>' in str(sys.stdin))
+        #self.assertTrue('<stdout>' in str(sys.stdout))


Can we uncomment these?

slozier · 2017-05-29T13:51:04Z

-    AreEqual(id(x), id(True))
-
-def test_str_dict():
-    print "CodePlex Work Item 13115"


Maybe preverse the codeplex # as a comment?

slozier · 2017-05-29T20:54:34Z

-if options.RUN_TESTS: #TODO - bug when generating Pydoc
-    from Merlin.Testing.FieldTest import *
-    from Merlin.Testing.TypeSample import *
+class FieldMiscTest(IronPythonTestCase):


We probably want skipunless is_cli on all of these.

I thought of that too, but wasn't sure we would ever try running these under CPython.

I'm actually thinking about running the tests under CPython. Maybe not under CI, but it would be useful to highlight compatibility issues.

slozier · 2017-05-30T13:12:03Z


    def test_eval_dicts(self):
        # eval referencing locals / globals
+        import System


Maybe move the import to the if is_cli block?

slozier · 2017-05-30T13:13:06Z

        self.assertRaises(TypeError, tuple.__new__, str)
        self.assertRaises(TypeError, tuple.__new__, str, 'abc')

-    def test_override_pow(self):


Something wrong with this test?

slozier · 2017-05-31T15:01:16Z

@@ -1,125 +0,0 @@
-#####################################################################################


Why remove this test?

It was just calling the other loadorder tests, which will be run automatically now.

slozier · 2017-05-31T15:02:31Z

-    AreEqual(NSwForwardee1.Foo.A, 120)
-    AreEqual(NSwForwardee1.Bar.A, -120)
-
-@skip("multiple_execute", "posix")


In other places you kept the multiple_execute as a TODO however it seems you zapped it here.

Should be fixed now

slozier · 2017-05-31T15:13:21Z

+
+class BinasciiTest(unittest.TestCase):
+
+    @skipUnlessIronPython


I think we can remove the skipUnless, seems to work with CPython.

slozier · 2017-05-31T15:15:27Z

@@ -1,944 +0,0 @@
-#####################################################################################


Why remove this one?

It's renamed in a coming commit.

slozier · 2017-05-31T15:21:26Z

+        t_in.close()
+
+        self.assertEqual(len(t_out_lines), 0)
+        print t_err_lines[0]


Are these prints for testing?

Yes, will remove

…date more tests

slozier · 2017-06-01T14:12:50Z

                try {
                    byte[] buffer = new byte[2048];
-                    MemoryStream result = new MemoryStream(n);
+                    MemoryStream result = new MemoryStream(len);


Unrelated to the change, but wow, we're really allocating a 2gb array if len is not specified?

Yes, that is indeed bad...

slozier · 2017-06-01T14:24:44Z

-                 "<type 'socket.ssl'>")
-
-    #--Negative
-    if is_cpython:


Dropped the negative case?

No, the file was renamed to match the test_*.py naming convention

I meant the positive case of test_SSLType appears in test__ssl.py but I don't see the negative case...

Ah, that's because there is no SSLType for the _ssl module anymore. Both IronPython and CPython get a message saying the module object has no attribute SSLType, so the test was bogus.

slozier · 2017-06-01T14:27:32Z

+        s.connect((SSL_URL, SSL_PORT))
+        ssl_s = real_ssl.sslwrap(s._sock, False)
+
+        if not is_cli:


Any reason why the shutdown is excluded here but not in other tests?

No, I'll fix it

slozier · 2017-06-01T14:31:14Z

+        f.bufsize = 4096
+        self.assertEqual(4096, f.bufsize)
+
+    @unittest.skipIf(is_posix, 'some issue on mono, need to debug further')


Right, good catch.

slozier · 2017-06-01T14:41:47Z

+        nt.mkdir("dir_create_test")
+        try:
+            nt.mkdir("dir_create_test")
+            self.assertTrueUnreachabale("Cannot create the same directory twice")


assertTrueUnreachable?

replace all issue, will fix

slozier · 2017-06-02T14:39:34Z

+
+    def test_negTestFunc_ObjectSubClass_testFunc_pw_kw_dupArg(self):
+        o = ObjectSubClass()
+        try: o.testFunc_pw_kw(a='abc')


Any idea why assertRaises doesn't work for these? Maybe add a comment that assertRaises should not be used here?

I think it may have something to do with how the parameter is passed around (in the case of NOT using assertRaises, it is setting the value of the 'a' parameter, but when passed into assertRaises it is passed as **kwArgs. A comment is a good idea

C Python doesn't seem to have any issues with it though, so I guess we have a bug.

Unit test updates

87a4720

slozier reviewed May 26, 2017

View reviewed changes

More tests passing

0a0c8eb

slozier reviewed May 26, 2017

View reviewed changes

slide added 4 commits May 26, 2017 15:29

More tests passing

e2d2092

try adding groups.io address

7a200b3

More tests passing

87720bb

More tests refactored

ea5b8d7

slozier reviewed May 29, 2017

View reviewed changes

More refactoring

7e0a6fe

slozier reviewed May 29, 2017

View reviewed changes

More tests passing

80fb305

slozier reviewed May 30, 2017

View reviewed changes

slide added 2 commits May 30, 2017 14:06

More tests refactored

3c9feea

More tests fixed and files removed

25a5062

slozier reviewed May 31, 2017

View reviewed changes

slide added 3 commits May 31, 2017 17:03

Switch implementation of skipUnlessIronPython, add retryOnFailure, up…

eaf3bab

…date more tests

Linux build

d479fdf

More tests refactored

1166576

slozier reviewed Jun 1, 2017

View reviewed changes

slide added 3 commits June 1, 2017 10:04

More fixes

cb893cd

fixes

d0b997e

Update PROCESS isolation level

abd3ddb

slozier reviewed Jun 2, 2017

View reviewed changes

slide merged commit 0fe44c3 into IronLanguages:master Jun 2, 2017

slide deleted the ironpython_unittest_updates branch June 2, 2017 19:45

slide changed the title ~~[WIP] Unit test updates~~ Unit test updates Jun 2, 2017

slide mentioned this pull request Nov 25, 2017

Updates for tests from ipy2 IronLanguages/ironpython3#353

Merged

slozier mentioned this pull request Oct 27, 2018

Update test_range to new style IronLanguages/ironpython3#469

Closed


		from iptest import IronPythonTestCase, is_cli

		@unittest.skipUnless(is_cli, 'IronPython specific test case')

		@@ -1,142 +0,0 @@
		#####################################################################################

		Fail("Returned value of getdebug method is not valid value:" + str(result))


		#CodePlex Work Item# 8202

		@@ -1,125 +0,0 @@
		#####################################################################################

		@@ -1,944 +0,0 @@
		#####################################################################################

Conversation

slide commented May 25, 2017

Uh oh!

slozier left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

slide May 26, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

slide May 26, 2017 •

edited

Loading