Repository navigation
Expand file tree
/
Copy pathdebugging.html
More file actions
780 lines (777 loc) · 70.6 KB
/
Copy pathdebugging.html
File metadata and controls
780 lines (777 loc) · 70.6 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<html><head><title>Python: package debugging</title>
</head><body bgcolor="#f0f0f8">
<table width="100%" cellspacing=0 cellpadding=2 border=0 summary="heading">
<tr bgcolor="#7799ee">
<td valign=bottom> <br>
<font color="#ffffff" face="helvetica, arial"> <br><big><big><strong>debugging</strong></big></big></font></td
><td align=right valign=bottom
><font color="#ffffff" face="helvetica, arial"><a href=".">index</a><br><a href="file:e:\vslprf\dlr\languages\ironpython\tests\debugging\__init__.py">e:\vslprf\dlr\languages\ironpython\tests\debugging\__init__.py</a></font></td></tr></table>
<p><tt>-------------------------------------------------------------------------------<br>
INTRODUCTION<br>
This test plan addresses debugging test coverage for the 2.x series of <br>
IronPython interpreter releases.<br>
<br>
EDITORS NOTES<br>
Throughout this document you will find references to documentation in<br>
other packages similar to "See documentation for debugging.pdb_module". This simply<br>
means that relative to this package (i.e., debugging), you should follow<br>
the 'pdb_module (package)' link at the bottom of this page.<br>
<br>
-------------------------------------------------------------------------------<br>
FEATURE GOALS<br>
The goals of IP's debugging feature is to provide IronPython users with <br>
complete use of CPython's existing debugger functionality, the "pdb" and<br>
"bdb" modules respectively, while also providing a great interactive debugging <br>
experience using Microsoft tools such as mdbg and the Visual Studio IDE.<br>
<br>
-------------------------------------------------------------------------------<br>
PRIMARY TESTING CONCERNS<br>
- new builtin module attributes added to IP to support CPython's "bdb" and "pdb"<br>
modules<br>
- compatibility with CPython's use of the "bdb" and "pdb" modules<br>
- what happens breakpoint-wise when IP steps into .NET code from "pdb"?<br>
- .NET applications hosting IP should be able to debug Python code<br>
- the experience of debugging IP under Microsoft's mdbg and Visual <br>
Studio tools is what existing CSharp and VB.Net would expect; but still<br>
Pythonic<br>
- "-D" command-line flag<br>
<br>
-------------------------------------------------------------------------------<br>
PRIMARY TESTING FOCUS<br>
Our principal testing focus will be on testing that IP's support of the<br>
"pdb" module is completely compatible with that of CPython by running existing<br>
CPython tests, and also by creating a new generic side-by-side testing framework.<br>
<br>
The secondardy focus will be testing that debugging IP under mdbg is a<br>
pleasant experience. This will initially be acheived by using the CLR Team's<br>
existing debugging test framework, but in the long run we will move to a<br>
new generic, side-by-side testing framework which can be reused for other <br>
side-by-side testing areas.<br>
<br>
-------------------------------------------------------------------------------<br>
REFERENCES<br>
* IronPython Test Plan Internal IP SharePoint site<br>
* Feature Specifications None...<br>
* Development Documents None...<br>
* Schedule Documents None...<br>
* CPython pdb module <a href="http://docs.python.org/library/pdb.html">http://docs.python.org/library/pdb.html</a><br>
* CPython bdb module <a href="http://docs.python.org/library/bdb.html">http://docs.python.org/library/bdb.html</a><br>
* pdb tutorial <a href="http://onlamp.com/pub/a/python/2005/09/01/debugger.html">http://onlamp.com/pub/a/python/2005/09/01/debugger.html</a><br>
* MDbg.exe <a href="http://msdn.microsoft.com/en-us/library/ms229861.aspx">http://msdn.microsoft.com/en-us/library/ms229861.aspx</a><br>
* Debugging IP in Visual Studio <a href="http://devhawk.net/2008/05/08/Debugging+IronPython+Code+In+Visual+Studio.aspx">http://devhawk.net/2008/05/08/Debugging+IronPython+Code+In+Visual+Studio.aspx</a><br>
* Bug queries <a href="http://ironpython.codeplex.com/WorkItem/AdvancedList.aspx">http://ironpython.codeplex.com/WorkItem/AdvancedList.aspx</a><br>
and set the component to "Debugging"<br>
* Build drop location Updated at <a href="http://ironpython.codeplex.com/SourceControl/ListDownloadableCommits.aspx">http://ironpython.codeplex.com/SourceControl/ListDownloadableCommits.aspx</a><br>
on weekdays<br>
* source file tree $/.../dlr/Debugging/ and also $/.../dlr/.../IronPython/*<br>
<br>
-------------------------------------------------------------------------------<br>
PERSONNEL<br>
Program Manager: hpierson<br>
Developer: dinov<br>
Tester: dfugate<br>
<br>
-------------------------------------------------------------------------------<br>
TESTING SCHEDULE<br>
<br>
Phase I - Test Plan and Enabling Existing Tests<br>
- Write test plan for IronPython Debugging feature<br>
- Enable any existing CPython test cases for "pdb" and "bdb" in SNAP<br>
- Enable any existing CPython test cases for Python features supporting <br>
"pdb" and "bdb" in SNAP<br>
<br>
Phase II - Examine Coverage of Existing Cases and Write High-priority Cases<br>
- Verify that developer tests for new additions to IP <br>
supporting "pdb" and "bdb" modules are sufficient. Write more test<br>
cases if not<br>
- Write additional test cases for the "pdb" module if CPython's test cases <br>
are insufficient<br>
- Examine our existing mdbg test (cases) looking for holes. Write new test<br>
cases if needed<br>
- Author a few manual sanity test cases for verifying IP can <br>
be debugged under the Visual Studio IDE. These should be folded into<br>
the CodePlex test pass signoff procedure for major releases<br>
<br>
Phase III - Design and Implement Side-by-Side Testing Framework<br>
- Based on the needs of test cases added in the previous phase either:<br>
1. Design a side-by-side testing infrastructure to be used with testing<br>
IP's CPython debugging compatibility OR<br>
2. Look for existing side-by-side testing frameworks we could reuse.<br>
The pexpect package might be useful here<br>
- Implement (or add existing) side-by-side testing framework<br>
<br>
Phase IV - Implement High-priority Cases<br>
- Implement test cases for additions to IP which support the<br>
"pdb" module IF NEEDED<br>
- Implement test cases for the "pdb" module IF NEEDED<br>
- Implement test cases for mdbg IF NEEDED<br>
<br>
Phase IV - Write Lower Priority Cases<br>
- Write test cases around stepping into (hosted) Python code from a <br>
.NET application<br>
- Write test cases around stepping into .NET code from an ipy.exe<br>
process<br>
- Write performance test cases<br>
<br>
Phase V - Implement Lower Priority Cases<br>
- Implement test cases around stepping into (hosted) Python code from a <br>
.NET application<br>
- Implement test cases around stepping into .NET code from an ipy.exe<br>
process<br>
- Implement performance test cases<br>
<br>
Phase VI - Future<br>
- Develop test cases for the "bdb" module if CPython's test cases are<br>
insufficient<br>
- Automate manual sanity Visual Studio debugging tests<br>
- Migrate existing mdbg tests to the new side-by-side testing framework<br>
- Low-resource test cases and implementation<br>
- Reliability test cases and implementation<br>
- Scalability test cases and implementation<br>
- Stress test cases and implementation<br>
- Internationalization test cases and implementation<br>
<br>
Phase P - All Public Releases<br>
- Verify all user documentation with respect to debugging<br>
- Setup testing - verify pdb.py and all supporting modules are installed by the MSI<br>
- Generate code coverage report and verify block coverage as greater<br>
than 80%<br>
<br>
-------------------------------------------------------------------------------<br>
FEATURE HISTORY<br>
2.0: Tests implemented for IronPython under mdbg. Scenarios which still <br>
need to be covered are:<br>
* Attach scenario with and without the -D option<br>
* Step into/out when there is no pdb file for IP <br>
binaries<br>
* Multiple statements in one line (e.g., "x = 1;y=2"<br>
* if x and \n y: print 1<br>
* dict/tuple spanning multiple lines<br>
* break/continue, explicit raise<br>
* other multi-line statement scenarios such as:<br>
if True: print 1<br>
if True:\n print 1<br>
* MdbgInterop<br>
able to break into the C# code from Python<br>
able to break back into Python code from C#<br>
<br>
2.6B2: First release of IronPython to include support of CPython's pdb module.<br>
Only works if -X:Frames or -X:FullFrames option supplied to ipy.exe.<br>
No pdb-related tests running in SNAP at this point, but we are running<br>
all of CPython's tests WRT supporting functions in the sys module.<br>
Debugging test plan completely rewritten in the form of pydoc strings,<br>
and made public.<br>
<br>
-------------------------------------------------------------------------------<br>
FEATURES:<br>
- sys.settrace, sys.gettrace, and sys.call_tracing. NOTE: while each of these<br>
three builtin module members needs to be tested within unit tests on their<br>
own, the real use case here is through the "bdb" and "pdb" modules<br>
- ipy.exe's generation of PDBs<br>
- ipy.exe's PDB visualizers<br>
<br>
-------------------------------------------------------------------------------<br>
FILES AND MODULES:<br>
<br>
- FILES LIST:<br>
* everything under Microsoft.Scripting.Debugging, although the DLR <br>
officially owns this<br>
* sys.cs<br>
* IronPython.Runtime.PythonTracebackListener<br>
- REGISTRY, INI SETTINGS: None<br>
- SETUP PROCEDURES: <br>
CPython standard library must be installed and present in sys.path to be <br>
usable<br>
- DE-INSTALLATION PROCEDURES<br>
N/A<br>
- DATABASE SETUP AND PROCEDURES<br>
N/A<br>
- NETWORK DOMAIN/TOPOLOGIES CONFIGURATION PROCEDURES<br>
N/A<br>
- PERFORMANCE MONITORING COUNTERS SETUP AND CONFIGURATIONS<br>
N/A<br>
<br>
-------------------------------------------------------------------------------<br>
OPERATIONAL ISSUES<br>
N/A. This feature of IP is not being monitored/maintained by operational <br>
staff, and IP is provided on an as-is basis.<br>
<br>
-------------------------------------------------------------------------------<br>
SCOPE OF TEST CASES<br>
We'll be getting most of our test coverage on the debugging feature of <br>
IP by unit testing the CPython 'pdb' module. Doing this thoroughly<br>
should guarantee we hit nearly all blocks of code supporting debugging in <br>
IP DLLs. This will be validated and verified through monthly code <br>
coverage runs. As an aside, this should in theory hit most of the DLR's<br>
Microsoft.Scripting.Debugging.dll as well.<br>
<br>
As IP currently has no tie-in into the Visual Studio IDE, this is <br>
arguably far less important to test than our support of the 'pdb' module or <br>
even mdbg. As a result, the Microsoft toolset testing emphasis will involve<br>
exahaustively covering all aspects of mdbg. That said, some of our users do in<br>
fact have VS Pro installed implying that we should perform at least minimal,<br>
manual sanity tests before every major public release of IP. Should we<br>
ever provide visualizers for our debug symbols we'll need much more <br>
comprehensive and automated tests for VS of course.<br>
<br>
-------------------------------------------------------------------------------<br>
ACCEPTANCE CRITERIA<br>
- no debugging related feature should be checked into the internal source <br>
repository without some form of unit test hooked into SNAP checked in as<br>
well<br>
- we should not publically advertise the existance of any new debugging features<br>
in an IP release unless:<br>
- if the feature is CPython-based; the majority of it's corresponding <br>
CPython unit test cases have been enabled in SNAP and pass<br>
- if the feature is novel to IP; block coverage should be greater<br>
than 70%<br>
- the feature needs to be documented in some form other than a blog<br>
- performance of the feature needs to be "within reason"<br>
- if the feature is not CPython-based; it needs a Pythonic feel to it<br>
- at least half of all bugs opened on the feature since its inception are<br>
fixed<br>
<br>
-------------------------------------------------------------------------------<br>
KEY FEATURE ISSUES<br>
- no spec on debugging IP from mdbg, VS, or what debugging symbols <br>
should be generated for various Python constructs<br>
- insufficient test resources to adequately test this feature<br>
- insufficient resources to implement all debugging features called out in this<br>
test plan<br>
- existing mdbg tests are based on CLR infrastructure and require a Perl <br>
installation. We cannot redistribute this publically<br>
- existing mdbg tests in SNAP are flakey<br>
<br>
-------------------------------------------------------------------------------<br>
TEST APPROACH<br>
- DESIGN VALIDATION<br>
We have no part in the design of the "pdb" and "bdb" standard modules. <br>
<br>
As for debugging under mdbg...there is no current design document on <br>
debugging Python sessions. Generally speaking test will simply ensure that<br>
whatever mdbg functionality exists with regard to IP is "Pythonic",<br>
but at the same time familiar to CSharp and VB developers.<br>
<br>
- DATA VALIDATION<br>
pdb/bdb: at a very low-level, all we need to validate is that parameters passed<br>
to a trace function we set via sys.settrace are as expected. In particular,<br>
the first parameter passed in is the current stack frame which we have a <br>
very solid chance of getting wrong.<br>
<br>
mdbg/VS: need to validate that generation of PDB symbols "make sense"<br>
with respect to whatever Python source is being compiled. There are at <br>
least two ways to accomplish this:<br>
1. Create a regression test in which we compare the generated PDB of a <br>
known Python script to an expected PDB file. The Python script should<br>
exhaustively cover everything that's possible in the Python grammar.<br>
Testing in this manner is not really maintainable in the long run as we<br>
expect the PDB generated for a given Python script to change over time<br>
reflecting optimizations and new features added to IP<br>
2. Test that the behavior of mdbg, given a known Python script, does not <br>
regress. Again, this Python script should exhaustively cover everything<br>
that's possible in Python's grammar, but this time around we'll also<br>
need to cover everything that's possible under mdbg<br>
<br>
- API TESTING<br>
We'll exhaustively cover the entire "pdb" standard module without ever <br>
touching the CSharp APIs directly. Prior coverage runs have shown that <br>
(generally speaking) any block of code that is not directly hittable through <br>
the Python API is likely dead code. Also, testing "bdb" module is lower <br>
priority as "pdb" uses this for its own implementation. The final <br>
justification for this is that there is actually very little in terms of <br>
debugging APIs in IP DLLs. Most of the debugging support is built<br>
directly into the DLR's Microsoft.Scripting.Debugging.dll.<br>
<br>
For mdbg/VS, we'll exhaustively cover all mdbg commands.<br>
<br>
- CONTENT TESTING<br>
All debugging documentation should be thoroughly reviewed by Test before any<br>
public release of IronPython. Careful attention will be given to duplicating<br>
exactly any command fed into ipy.exe or mdbg.exe, and this should be <br>
automated entirely if feasible.<br>
<br>
- LOW-RESOURCE TESTING<br>
Use of the debugging feature has a few side effects which might be interesting<br>
from a limited resources perspective. First and foremost, use of the -X:Frames<br>
and -X:FullFrames IP console flags imply that IP will consume more <br>
memory than under normal circumstances. We should check that IP does not <br>
consume "too much" extra memory. Next, use of the debugging flag, '-D', in<br>
conjunction with the '-X:SaveAssemblies' flag will generate <br>
Snippets.debug.scripting.pdb in the %TMP% directory. What occurs if the drive<br>
containing %TMP% is full?<br>
<br>
NOTE: while this is an interesting test area to explore, there are no current<br>
plans to test it. We should perform general low-resource testing of <br>
IP before tackling it for this specific feature.<br>
<br>
- SETUP TESTING<br>
The only option of IP's MSI installer capable of affecting<br>
the debugging feature is the ability to selectively install CPython's standard<br>
library. As we generate the list of CPython modules to include in the MSI<br>
dynamically, it would be worthwhile to always check that pdb.py is included.<br>
<br>
NOTE: IronPython currently has no automated tests for the MSI. Once we<br>
do, we need to add a check to ensure pdb.py is always included. For the time<br>
being this will have to be done manually.<br>
<br>
- MODES AND RUNTIME OPTIONS<br>
We should focus our efforts on testing IP with the following modes:<br>
* -D -X:Frames<br>
* -D -X:FullFrames<br>
* -D -X:Frames -X:SaveAssemblies<br>
* -D<br>
* -X:Frames<br>
* <br>
using both debug and release IP assemblies. Also note that this<br>
needs to be performed for both Python interactive sessions and Python scripts<br>
passed into ipy.exe (e.g., "ipy.exe test_str.py"). Last but<br>
not least, there's the %PYTHONDEBUG% environment variable to consider.<br>
<br>
- INTEROPERABILITY<br>
Of primary concern is that IP produces an identical experience as<br>
CPython when given input which utilizes the pdb module. This will be <br>
accomplished via side-by-side testing and verification that the output of <br>
IP is the same as CPython given an identical input.<br>
<br>
We will also verify that IP can be debugged from mdbg and the Visual<br>
Studio IDE.<br>
<br>
- INTEGRATION TESTING<br>
* Silverlight support of debugging?<br>
* breaking into .NET code from pdb<br>
* breaking into COM from pdb<br>
* breaking into IronRuby from pdb<br>
* breaking into Python code from DLR hosting APIs<br>
* breaking into Python code from IronRuby<br>
<br>
- BETA TESTING<br>
The second beta release of IronPython 2.6 will support the debugging feature<br>
in the form of very limited pdb module support with the -X:Frames flag <br>
passed to ipy.exe. As with all beta releases of IP, this will be <br>
released on CodePlex to the general public. We will then use feedback<br>
from the IronPython Community to determine the amount of effort that goes <br>
into fully testing this feature.<br>
<br>
- ENVIRONMENT/SYSTEM - GENERAL<br>
The %PYTHONDEBUG% environment variable may have some impact on this feature.<br>
Other than this, the VS IDE will of course need to be installed for testing<br>
the VS debugging experience.<br>
<br>
- CONFIGURATION<br>
Need .NET 2.0 Service Pack 1 installed to run IP, and .NET 3.5<br>
Service Pack 1 to build the feature.<br>
<br>
- USER INTERFACE<br>
IP provides no user interfaces.<br>
<br>
- PERFORMANCE & CAPACITY TESTING<br>
Minimally, we need to run one or more tests in the perf lab under the -X:Frames<br>
and -X:FullFrames test modes. We should also measure the end-to-end run time<br>
of a complete, yet minimal, debugging sample utilizing the pdb module in the <br>
lab. The focus here will simply be on ensuring there are no perf regressions<br>
which are greater than 15% for any given checkin, and also that overall <br>
performance of the debugging feature remains palatable to our userbase.<br>
<br>
In terms of capacity testing we should look at:<br>
- if perf gets affected when breaking into deeply nested functions (e.g., <br>
recursive functions)<br>
- anything else?<br>
<br>
- PRIVACY<br>
Does %TMP%\Snippets.debug.scripting.pdb contain sensitive customer data? If <br>
so, does IP disclose the existence of this file?<br>
Is remote debugging supported? If so, what safeguards are in place?<br>
<br>
- RELIABILITY<br>
Most end-users are expected to enter into debugging sessions for short <br>
periods of time to diagnose issues with Python code. From this perspective<br>
the so-called "up time" of debugging sessions is less important than the <br>
overall sporadic failure frequency. We can get a good sense of how often <br>
sporadic failures occur by automatically running debugging tests in SNAP <br>
for every IP checkin.<br>
<br>
- SCALABILITY<br>
Does debugging still work correctly when stepping into deeply nested (e.g., 1000)<br>
functions?<br>
<br>
- STRESS TESTING<br>
- gcstress<br>
- MDA<br>
- assemblies in the GAC<br>
- call pdb functions multiple times and look for memory leaks<br>
- what happens when stepping into unreasonably deeply nested functions?<br>
<br>
- VOLUME TESTING<br>
App building exercise or try utilizing pdb from some of our internal Python <br>
tools. It would be interesting to use this from gopackage when exceptions <br>
are encountered. <br>
<br>
- INTERNATIONAL ISSUES<br>
As Python is not a localized language and available only in English, we <br>
simply need to confirm that nothing gets localized. This can be done by<br>
running a simple debugbing test against non-English OSes such as Deutsche<br>
Vista.<br>
<br>
- ROBUSTNESS<br>
We'll need a dedicated test or two in the stress lab to ensure there are no<br>
memory leaks. Also, we'll need to keep an eye out for sporadic failures of<br>
debugging tests in SNAP.<br>
<br>
- ERROR TESTING<br>
What happens when invalid commands are fed to the pdb debugger? Is this <br>
handled identically to CPython?<br>
<br>
Is there any conceivable way to fully break debugging without actually <br>
breaking ipy.exe in general? What happens under the mode <br>
"-D -X:SaveAssemblies" when the current user doesn't have write <br>
permissions on %TMP%? What happens if MS.Scripting.Debugging.dll is <br>
removed outright?<br>
<br>
- USABILITY<br>
There are three major usability issues with debugging:<br>
* IP offers no visualizer for the VS IDE<br>
* we're largely incompatible with CPython's pdb module<br>
* debugging Python code from .NET apps isn't currently possible<br>
<br>
Test will depend on feedback from the IronPython Mailing List and blogs to<br>
determine more usability issues. Resource permitting, we may also do <br>
app building exercises.<br>
<br>
The usability goal of this feature is that there are no complaints about<br>
the IP debugging experience from our users.<br>
<br>
- ACCESSIBILITY<br>
As this feature provides no new user interfaces to IronPython. and simply <br>
emits output to the stdout stream of command prompts, there should be no <br>
need to invest into accessibility testing.<br>
<br>
- USER SCENARIOS<br>
It's anticipated that users of this feature will be limited almost exclusively to<br>
developers. Expert Python users will likely prefer using the pdb<br>
module or debugging their applications directly via the "-i" ipy.exe <br>
option. Existing Microsoft customers with little Python background will <br>
probably be more comfortable with the VS IDE debugging experience. Due to <br>
this we must make sure both experiences are great!<br>
<br>
The types of applications being debugged will likely:<br>
* be non-trivial<br>
* consist of many different modules/packages<br>
* make heavy use of the CPython standard library and/or third party Python<br>
packages<br>
* be debugged often during the application development phase<br>
* be debugged infrequently in a production environment when something goes <br>
awry<br>
<br>
With these constraints in mind, we should try to debug existing major third<br>
party Python applications and/or add debugging support to our own internal<br>
Python tools.<br>
<br>
- BOUNDARIES AND LIMITS<br>
What happens when stepping into a recursive function at the maximum recursion<br>
level?<br>
<br>
- OPERATIONAL ISSUES<br>
None<br>
<br>
- SPECIAL CODE PROFILING AND OTHER METRICS<br>
Overall block coverage of all assemblies with "ipy" or "IronPython" in their<br>
names should be above 80% and file coverage should stay above 97%. Visual <br>
Studio's mstest will be used to measure code coverage.<br>
<br>
-------------------------------------------------------------------------------<br>
TEST ENVIRONMENT<br>
<br>
- OPERATING SYSTEMS<br>
32-bit Windows XP<br>
32-bit Windows 2003<br>
64-bit Windows 2003<br>
32-bit Windows Vista<br>
64-bit Windows Vista<br>
32-bit Windows Vista (Deutsche)<br>
- NETWORKS<br>
General Intranet network connection required for SNAP.<br>
May be other special network needs if remote debugging is/becomes a supported scenario.<br>
- HARDWARE<br>
- MACHINES<br>
At least two machines of every OS variety called out above with at least<br>
10 gigs of free hard disk space, 2 gigs of RAM, and a modern CPU<br>
- SOFTWARE<br>
* Visual Studio 2008 Team System Service Pack 1 installed to run the <br>
test suite<br>
* PowerShell 1.0 to run any supporting test scripts<br>
* CPython for side-by-side tests<br>
* mdbg<br>
<br>
-------------------------------------------------------------------------------<br>
UNIQUE TESTING CONCERNS FOR SPECIFIC FEATURES<br>
It should be relatively easy to drive pdb or mdbg via the command line in an<br>
automated fashion, but the same cannot be said about the VS IDE debugging <br>
experience. We'll need to reuse the VS team's automation infrastructure and <br>
ensure any machines running these tests in SNAP are never locked at the <br>
username/password screen which would cause the test to fail. Also, should <br>
visualization ever be a supported scenario, we may have no other option than to <br>
test this manually (note - this needs to be investigated).<br>
<br>
-------------------------------------------------------------------------------<br>
AREA BREAKDOWN<br>
- CPython's "pdb" module<br>
See debugging.pdb_mod.<br>
<br>
- mdbg tool support<br>
See debugging.mdbg_tool.<br>
<br>
- VS IDE<br>
See debugging.vs.<br>
<br>
- sys module extensions<br>
- sys.settrace<br>
settrace(function) <br>
Set the system's trace function, which allows you to implement a Python <br>
source code debugger in Python. The function is thread-specific; for a <br>
debugger to support multiple threads, it must be registered using <br>
settrace() for each thread being debugged.<br>
<br>
Trace functions should have three arguments: frame, event, and arg. frame <br>
is the current stack frame. event is a string: 'call', 'line', 'return', <br>
'exception', 'c_call', 'c_return', or 'c_exception'. arg depends on the <br>
event type.<br>
<br>
The trace function is invoked (with event set to 'call') whenever a new <br>
local scope is entered; it should return a reference to a local trace <br>
function to be used that scope, or None if the scope shouldn't be traced.<br>
<br>
The local trace function should return a reference to itself (or to another <br>
function for further tracing in that scope), or None to turn off tracing <br>
in that scope.<br>
<br>
The events have the following meaning:<br>
'call'<br>
A function is called (or some other code block entered). The global <br>
trace function is called; arg is None; the return value specifies the <br>
local trace function.<br>
'line'<br>
The interpreter is about to execute a new line of code (sometimes <br>
multiple line events on one line exist). The local trace function is <br>
called; arg is None; the return value specifies the new local trace <br>
function.<br>
'return'<br>
A function (or other code block) is about to return. The local trace <br>
function is called; arg is the value that will be returned. The trace <br>
function's return value is ignored.<br>
'exception'<br>
An exception has occurred. The local trace function is called; arg is <br>
a tuple (exception, value, traceback); the return value specifies the <br>
new local trace function.<br>
'c_call'<br>
A C function is about to be called. This may be an extension function <br>
or a builtin. arg is the C function object.<br>
'c_return'<br>
A C function has returned. arg is None.<br>
'c_exception'<br>
A C function has thrown an exception. arg is None.<br>
<br>
Note that as an exception is propagated down the chain of callers, an <br>
'exception' event is generated at each level.<br>
<br>
- sys.gettrace<br>
Get the trace function as set by settrace().<br>
<br>
- sys.call_tracing<br>
call_tracing(func, args) -> object<br>
<br>
Call func(*args), while tracing is enabled. The tracing state is<br>
saved, and restored afterwards. This is intended to be called from<br>
a debugger from a checkpoint, to recursively debug some other code.<br>
<br>
-------------------------------------------------------------------------------<br>
TEST CASE STRUCTURE<br>
Test cases will be stored directly in the pydoc strings of the functions <br>
implementing the tests. In turn, the modules containing these test case <br>
functions can be found under the "debugging" test package. Generating pydoc for<br>
the "debugging" test package and all sub-modules/sub-packages will generate <br>
this test plan.<br>
<br>
-------------------------------------------------------------------------------<br>
SPEC REVIEW ISSUES<br>
There is no IronPython spec for this feature. Any issues observed in the pdb <br>
module, which acts as a spec, and reproducible under CPython should be reported <br>
to <a href="http://bugs.python.org">http://bugs.python.org</a>.<br>
<br>
-------------------------------------------------------------------------------<br>
TEST TOOLS<br>
One of the following:<br>
- reuse CLR's debugging test infrastructure. NOTE: this is only suitable for <br>
mdbg tests<br>
- write our own new side-by-side testing infrastructure entirely in Python<br>
so that we can eventually contribute pdb tests back to CPython<br>
- write our own new side-by-side testing infrastructure in any<br>
language using any framework. <a href="http://sourceforge.net/projects/pexpect/">http://sourceforge.net/projects/pexpect/</a> could<br>
be useful as would be PowerShell<br>
- re-use some internal tool that might not be redistributable on CodePlex<br>
<br>
which will minimally need to support the following features:<br>
- ability to ignore lines (matching some regular expression) entirely<br>
- ability to substitute strings in lines<br>
- ability to run arbitrary commands (e.g., not limited to "ipy.exe ..." or "mdbg ...")<br>
- ability to match the current output against a reference output stored on disk<br>
- ability to run two or more arbitrary commands and compare their output against each other<br>
<br>
As per usual, the plan is to automate all tests using the SNAP checkin system<br>
and VS's mstest tool.<br>
<br>
-------------------------------------------------------------------------------<br>
SMOKE TEST (ACCEPTANCE TEST, BUILD VERIFICATION, ETC.)<br>
<br>
pdb acceptance test<br>
ipy.exe -c "import pdb"<br>
<br>
mdbg<br>
TODO (July 30, 2009)<br>
<br>
Visual Studio IDE<br>
TODO (July 30, 2009)<br>
<br>
<br>
-------------------------------------------------------------------------------<br>
AUTOMATED TESTS<br>
pdb and mdbg tests will be 100% automated in SNAP.<br>
<br>
<br>
-------------------------------------------------------------------------------<br>
MANUAL TESTS<br>
Setup testing will be manual for now, and we'll eventually move to automated <br>
testing in SNAP using PowerShell support scripts.<br>
<br>
Visual Studio IDE testing is also manual for now. Sometime in the <br>
future we'll migrate to using the VS team's automation technologies.<br>
<br>
Manual testing will only occur for major IP releases and the <br>
instructions will be documented on the internal IP website under the <br>
"Release Process" wiki.<br>
<br>
<br>
-------------------------------------------------------------------------------<br>
REGRESSION TESTS<br>
Regression tests will be added to existing test modules under the "debugging" <br>
package before any CodePlex or internal bug dealing with debugging is closed.<br>
As such, the majority of these regressions will be automated and run on every<br>
developer or test checkin. Developers should preferably add a regression test<br>
for every bug fix, and Test will verify the sufficiency of the test, extending<br>
it if necessary.<br>
<br>
-------------------------------------------------------------------------------<br>
BUG BASHES<br>
While considered to be extremely useful, we have no current plans for internal <br>
bug bashes on this feature. Quite simply put, we need more IronPython headcount <br>
and/or interest from the rest of Visual Studio Languages to accomplish this.<br>
<br>
-------------------------------------------------------------------------------<br>
BUG REPORTING<br>
All bug reports on this feature which do not include information consided to be<br>
Microsoft confidential are to be filed at <br>
<a href="http://ironpython.codeplex.com/WorkItem/AdvancedList.aspx">http://ironpython.codeplex.com/WorkItem/AdvancedList.aspx</a> under the "Debugging"<br>
component using the bug template found at <br>
<a href="http://ironpython.codeplex.com/Wiki/View.aspx?title=IronPython%20Bug%20Template">http://ironpython.codeplex.com/Wiki/View.aspx?title=IronPython%20Bug%20Template</a>.<br>
Bugs will be triaged on a weekly basis by the entire IronPython Team.<br>
<br>
-------------------------------------------------------------------------------<br>
PLAN CONTINGENCIES<br>
Without at least one additional SDET or other test tasks getting dropped, only <br>
phases I and II of this plan are implementable prior to the release of <br>
IP 2.6 RTM.<br>
<br>
-------------------------------------------------------------------------------<br>
EXTERNAL DEPENDENCIES<br>
We depend on the DLR for their implementation of Microsoft.Scripting.Debugging.dll <br>
which forms the basis of our support of the CPython pdb module. This plan <br>
assumes that this DLL has been adequately tested.<br>
<br>
There are no teams/projects we know about which are taking a dependency on <br>
IP's debugging functionality.<br>
<br>
-------------------------------------------------------------------------------<br>
HEADCOUNT REQUIREMENTS<br>
At least two FTE IronPython SDETs will be needed to implement this test plan <br>
fully: one to handle general, day-to-day IP test operations<br>
(i.e., passes/investigations/issues/regressions, etc), and another to work exclusively on the <br>
debugging feature. A loose approximation is that the SDET working full-time<br>
on debugging would need anywhere from one to three months to completely <br>
implement all aspects of this plan. This estimate varies considerably <br>
depending upon how much emphasis is to be given to pdb versus mdbg versus the <br>
VS IDE. Having a second full-time SDET on the debugging feature would have a<br>
not positive effect as the division of labor between testing command-line IP<br>
debugging support and VS IDE support is quite clear.<br>
<br>
-------------------------------------------------------------------------------<br>
PRODUCT SUPPORT<br>
IP is not a supported product in the conventional Microsoft sense. <br>
Support for IP is freely given directly by the IP Team via the <br>
IronPython Mailing List.<br>
<br>
-------------------------------------------------------------------------------<br>
DROP PROCEDURES<br>
All developer checkins must go through the SNAP checkin system => IronPython<br>
sources are always in a "good" state. Test will build IP from sources<br>
using developer instructions.<br>
<br>
-------------------------------------------------------------------------------<br>
RELEASE PROCEDURES<br>
See the IP "Release Process" wiki on the internal IP website.<br>
<br>
-------------------------------------------------------------------------------<br>
ALIAS/NEWSGROUPS AND COMMUNICATION CHANNELS<br>
Major changes around IP debugging support should be announced on the <br>
<br>
-------------------------------------------------------------------------------<br>
REGULAR MEETINGS<br>
- FEATURE TEAM MEETINGS<br>
There is no feature team for debugging.<br>
- PROJECT TEST TEAM MEETINGS<br>
IronPython weekly team meeting. Day and time are subject to change.<br>
We'll triage debugging work items during this meeting and discuss IP <br>
debugging issues as necessary.<br>
- FEATURE TEAM TEST MEETINGS<br>
There is no feature test team for debugging.<br>
<br>
-------------------------------------------------------------------------------<br>
DECISION MAKING PROCEDURES<br>
Decisions on this feature will be driven by the following criteria:<br>
- Microsoft business needs<br>
- IronPython Community feature requests<br>
- Previous decisions made by CPython. E.g., the pdb module<br>
- Existing debugging support in Microsoft products such as VS<br>
In the event a conscensus cannot be reached by the team on something, Jim<br>
Hugunin's input should be sought.<br>
<br>
Miscellaneous Procedures:<br>
- the gopackage tool is responsible for validating all source pushes to <br>
CodePlex<br>
- at least two IP team members need to sanity check public builds<br>
before they can be released<br>
- full test pass signoff on this feature will be needed for every public, signed<br>
release<br>
- bug triages will be performed with all three software disciplines in attendance<br>
- development design should be reviewed both by the PM and Test disciplines<br>
<br>
-------------------------------------------------------------------------------<br>
NOTES<br>
<br>
MDBG TEST CASE WRITING TIPS<br>
- use different language structures as the last line in the Python file<br>
- when only one line lives inside a block of code, but the code should not be<br>
stepped through<br>
- correct step-through: no extra step-throughs or lack thereof<br>
-</tt></p>
<p>
<table width="100%" cellspacing=0 cellpadding=2 border=0 summary="section">
<tr bgcolor="#aa55cc">
<td colspan=3 valign=bottom> <br>
<font color="#ffffff" face="helvetica, arial"><big><strong>Package Contents</strong></big></font></td></tr>
<tr><td bgcolor="#aa55cc"><tt> </tt></td><td> </td>
<td width="100%"><table width="100%" summary="list"><tr><td width="25%" valign=top><a href="debugging.mdbg_tool.html"><strong>mdbg_tool</strong> (package)</a><br>
</td><td width="25%" valign=top><a href="debugging.pdb_mod.html"><strong>pdb_mod</strong> (package)</a><br>
</td><td width="25%" valign=top><a href="debugging.vs.html"><strong>vs</strong> (package)</a><br>
</td><td width="25%" valign=top></td></tr></table></td></tr></table>
</body></html>