Preface - Teasers - Enhanced Terminology - Reference - Encoding of DVD & Bluray Content - About Audio - Recovering The Camera Shots
Basic Primitives - Pulldown Primitives - Advanced Interpolations - Seen In The Wild, Repairing Video


Author comment: I submit these teasers in a brazen attempt to quickly hook the reader. Of course, I endeavor to follow the teasers with real -- perhaps even adequate -- documentation. Kindly bear in mind that the purpose of this documentation is to promote the notation, not the tricks herein and not potential implementations (though both tricks and potential implementations make interesting subtopics).

As a warmup, what follows is the notition for 2-3-2-3 pulldown that is probably familiar to everyone.

2-3-2-3 pulldown converts strides of 4 pictures at rate: #pps, to strides of 5 pictures at rate: (#x5/4)pps, to be used, for example, by cinema-at-30'fps.
import pictures: (A+a_______________)(B+b_______________)(C+c_______________)(D+d_______________)   ...#pps

NOTATION: #pps__#x2hps__(Aa-Dd)(AaBbBcCdDd)=(#x10/4)hps__(#x5/4)pps
           \          \  \                 \ \_____________________\___ weave
            \          \  \_________________\___ 2-3-2-3 pulldown
             \__________\___ unweave

export pictures: (A+a___________)(B+b___________)(B+c___________)(C+d___________)(D+d___________)   ...(#x5/4)pps
        combing:                                 XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
   1st halfpics: (A____-20%_____)(B____________+60%_____________)(C____-20%_____)(D____-20%_____)   ...[note 1]
   2nd halfpics: (a____-20%_____)(b____-20%_____)(c____-20%_____)(d____________+60%_____________)   ...[note 2]
Combing: 40% -- Judder: 80% @ 2 judders per stride [note 1][note 2].

[note 1] (A____-20%_____) is 20% shorter than (A_________________), (A____-20%_____)(B____________+60%_____________) is 80% judder @ 1 judder per stride with 1 2 1 1 cadence.

[note 2] (c____-20%_____) is 20% shorter than (c_________________), (c____-20%_____)(d____________+60%_____________) is 80% judder @ 1 judder per stride with 1 1 1 2 cadence.

Next, the notitions for 2 scan decombers that avoid bobbing. But first, some needed preparation regarding 'look' notation and the '§' operator and the '^' and '$' aliases. (Note that they are adapted excerpts of longer definitions in the "Reference" section.)

'look' notation & 'look' scanner: 'look' notation, with states: (look)(firstseen)(seen)(finalseen)(unseen), is a simple and straightforward scanning engine. It is the workhorse of the notation and can be nested. For example, a 'look' notation can be nested in (unseen) to thereby extend stream editing -- in fact, 'look' notations can also be nested in (firstseen) (seen) & (finalseen) to vastly extend stream editing. All import elements are processed in a single stream, so splitting streams is not required. If a (look) pattern is not 'seen' in the stream, (firstseen)(seen)(finalseen) are skipped and (unseen) is run. If no replacements are made in the import stream, it is exported without alteration. Otherwise, (seen) contains object patterns that can drop, add, reorder, and/or synthesize target frames (or pictures or halfpics or scans) without limitation. (firstseen) & (finalseen) operate like (seen) but they apply solely to scene changes -- thus, scene change is built into the architecture, not added as an afterthought to only selected functions.
Author comment: I apologize for making the outline above so short. 'look' notation is an object oriented, pattern detection & mapping process that's more fully documented in "Reference", "'look' notation & scanner". But I trust that the remainder of this teaser will expose 'look' notation's simplicity so that it's intuitively obvious to video-savvy readers.
§ operator: The section symbol: § , is an operator that interpolates, either temporally or spatially, 2 pictures or halfpics.

^ alias: The carat symbol: ^ , is an alias that 'reaches' beyond the current import stride to access the previous import stride. For example,
NOTATION: (Ab)(A A§b)(A ^§b)()
                     +-------1st stride------->+-------2nd stride------->+-------3rd stride------->
       NOTATION:     (A__________)(A§b________)(A__________)(^§b________)(A__________)(^§b________)..

import halfpics:     (A__________)(b__________)(C__________)(d__________)(E__________)(f__________)..   ...#hps
                      :`.¸¸¸¸¸¸¸¸  :            :            :            :            :                ...step 1: copy 'A' 'b' 'C' 'd' 'E' 'f' ..
                      :          `.`.           :            `.           :            `.
                     (A__________)(A§b________)(C__________)(^§d________)(E__________)(^§f________)..   ...#hps

import halfpics:     (A__________)(b__________)(C__________)(d__________)(E__________)(f__________)..   ...#hps
                                    `.¸¸¸¸¸¸¸¸¸¸¸¸¸¸¸¸¸¸¸¸¸   `.¸¸¸¸¸¸¸¸¸¸¸¸¸¸¸¸¸¸¸¸¸   `.¸¸¸¸¸¸¸¸¸¸    ...step 2: replace '^'s
                                                           `.                        `.
export halfpics:     (A__________)(A§b________)(C__________)(b§d________)(E__________)(d§f________)..   ...#hps
$ alias: The dollar symbol: $ , is an alias that 'reaches' beyond the current import stride to access the next import stride. For example,
NOTATION: (Ab)()(A§$ b)(A§b b)
                       +--next to final stride-->+------final stride------>
       NOTATION:     ..(A§$________)(b__________)(A§b________)(b__________)

import halfpics:     ..(W__________)(x__________)(Y__________)(z__________)   ...#hps
                        :            :            :    ¸¸¸¸¸¸.´:              ...step 1: copy ..'W' 'x' 'Y' 'z'
                        :            :            :  .´        :
                     ..(W§$________)(x__________)(Y§z________)(z__________)   ...#hps

import halfpics:     ..(W__________)(x__________)(Y__________)(z__________)   ...#hps
                             ¸¸¸¸¸¸¸¸¸¸¸¸¸¸¸¸¸¸¸.´                            ...step 2: replace '$'s
                           .´
export halfpics:     ..(W§Y________)(x__________)(Y§z________)(z__________)   ...#hps
With that preparation, what follows is the notitions for 2 scan decombers that avoid bobbing.

decomb-x2 decombs at the same resolution as bob, but without bobbing.
                       +-----first stride----->+------2nd stride------>    +-----final stride----->
import quasi pictures: (A+b___________________)(C+d___________________) .. (Y_________)(z_________)   ...#pps

NOTATION: #pps__#x2hps__(Ab)()(A§$ b)(A§b b)=#x2hps__#pps
           \          \  \                  \ \__________\___ weave
            \          \  \__________________\___ interpolate-x2 - (look)(firstseen)(seen)(finalseen)
             \__________\___ unweave

              unweave: (A_________)(b_________)(C_________)(d_________) .. (Y_________)(z_________)   ...#x2hps
       interpolate-x2: (B=A§C_____)(b_________)(D=C§E_____)(d_________) .. (Z=Y§z_____)(z_________)   ...#x2hps
                weave: (B+b___________________)(D+d___________________) .. (Z+z___________________)   ...#pps
Combing: none -- Judder: none -- Extra burden: 1 halfpic interpolation per stride -- Minor flaw: The final picture, only, is an interpolated blend of bobbed & combed.

Tip: At this point the reader should understand how '^' & '$' work, and how '(look)' '(firstseen)' '(seen)' & '(finalseen)' effects how the notation handles strides.

decomb-x2-hr is a high resolution version of decomb-x2 that 'looks' in both directions.
                        +-------first stride------->+--------2nd stride-------->    +-------final stride------->
 import quasi pictures: (A+b_______________________)(C+d_______________________) .. (Y+z_______________________)   ...#pps

NOTATION: #pps__#x2hps__(Ab)(A A§b A§$ b)(A ^§b A§$ b)(A ^§b A§b b)=#x4hps__#x2pps
           \          \  \                                         \ \____________\___ weave
            \          \  \_________________________________________\___ interpolate-x2 - (look)(firstseen)(seen)(finalseen)
             \__________\___ unweave

               unweave: (A___________)(b___________)(C___________)(d___________) .. (Y___________)(z___________)   ...#x2hps
        interpolate-x2: (A____)(a=A§b)(B=A§C)(b____)(C____)(c=b§d)(D=C§E)(d____) .. (Y____)(y=x§z)(Z=Y§z)(z____)   ...#x4hps
                 weave: (A+a_________)(B+b_________)(C+c_________)(D+d_________) .. (Y+y_________)(Z+z_________)   ...#x2pps
Combing: none -- Judder: none -- Extra burden: 2 halfpic interpolations per stride -- Minor flaw: The first & final pictures are interpolated blends of bobbed & combed.

What follows is an actual flawed video case study, its analysis, and a fix to transcode it into the best rendition possible given the source and its flaws.

"The Secret of SARAH SIDDONS" special feature in "ALL ABOUT EVE" Bluray, UPC 024543706878, is a forensic example of cinema-at-30'fps that was slightly wrecked by 2 mistakes.
Specimen: 'The Secret of SARAH SIDDONS', Lisa Orgolini interview, 'ALL ABOUT EVE', 00390.m2ts.png.
"The Secret of SARAH SIDDONS" is a 30'fps, made-for-TV documentary. It is a mix of 3 types of segments.
    _________________________________________________ "ALL ABOUT EVE" clips and interviews (cinema-at-24'fps-soft, nothing remarkable)
   /                                             /   ________________ pans & zooms, still photos (ersatz-NTSC, nothing remarkable)
  /                                             /   /            /   ________________________________________________ L.O. & C.M. interviews
 /                                             /   /            /   /                                            /
[24pps__48hps__(Aa-Dd)(AaBbBcCdDd)=60hps__30pps & 60'hps__30'pps & 24pps__48hps__(Aa-Dd)(AaBcCdDdDd)=60hps__30pps]30'fps
The problem is that 4 of the 21 interview segments: 3 Lisa Orgolini segments & 1 Chris Mankiewicz segment, are strange. They were probably added as an afterthought.
Analysis: Based on the cadence of the combed frames it is apparent that, like the other interviews, the L.O. & C.M. interviews began as 2-3-2-3 pulldown pictures.
                                                                          ___ camera shots
                                                                         /         _____________________ 2-3-2-3 pulldown to 30pps
                                                                        /         /                 /
                                                                  [24pps__48hps__(Aa-Dd)(AaBbBcCdDd)=60hps__30pps]30'fps
Next, the L.O. & C.M. interviews were probably converted from 30'fps to 24'fps -- why the conversion was done is unknown. In that conversion, instead of applying reverse 2-3-2-3 pulldown, picture Bb was dropped (and halfpic b was thereby lost).
                                                                                   ________________________ frame [B+b] dropped -- 1st mistake
                                                                                  /                    /
                                                            30'fps[30pps__60hps__(AaBbBcCdDd)(AaBcCdDd)=48hps__24pps]24'fps
Next, the L.O. & C.M. interviews were added to the almost finished "The Secret of SARAH SIDDONS", but the missing halfpic b created difficulty. At that point, the original camera shots for the L.O. & C.M. interviews must have no longer been available. Instead of bobbing the 1st halfpic, B, and then applying 2-3-2-3 pulldown, frame [D+d] was simply repeated.
                                                                                   ________________________ frame [D+d] repeated --  2nd mistake
                                                                                  /                    /
                                                            24'fps[24pps__48hps__(AaBcCdDd)(AaBcCdDdDd)=60hps__30pps]30'fps
Rehabilitation 1 -- What could have been done (and can still be done): Cut the L.O. & C.M. interviews, send then though the notation below, then paste them back into the video.
                                                                                                    ___________ bob 'B'
                                                                                                   //
                                                            30'fps[30'pps__60'hps__(AaBcCdDdDd)(AaBBBcCdDd)__30'pps]30'fps
Rehabilitation 2 -- A better approach: Run the entire video through the notation below. The odds are very high that only the L.O. & C.M. interviews will be matched and replaced. All other segments will be passed unchanged.
                            _____________ 'look' for the offending pattern
                           /         /__________________ the offending pattern 'seen'
                          /         //    __________/___ synthesize a new 'b' halfpic
                         /         //    / /       /
30'fps[30'pps__60'hps__(AaBcCdDdDd)(AaB a§c BcCdDd)__30'pps]30'fps
Rehabilitation 3 -- Even better: Run the entire video through the notation below to process it -- all 3 types of segments -- in a single pass -- no cut-&-paste. Bonus: The output is 120fps.
As a single line, the notation below is 209 characters and is difficult to read. Instead, labels: "NTSC?:" "FirstNTSC:" "NTSC:" "LastNTSC:" "NotNTSC:" "Damaged?:" "Fix:" "Pulldown?:" & "ReversePulldown:", can be used to compactly format the notation as follows:
30'fps[30pps__60hps__(NTSC?:)(FirstNTSC:)(NTSC:)(LastNTSC:)(NotNTSC:)]120fps
          NTSC?:AbCdEfGhIj     ...'look' for 5 pictures, no repeats (i.e. ersatz-NTSC)
      FirstNTSC:(Ab)(A A§b A§$ b)(A ^§b A§$ b)(           )=120hps__60pps__(A)(AA)=120pps
           NTSC:(Ab)             (A ^§b A§$ b)             =120hps__60pps__(A)(AA)=120pps
       LastNTSC:(Ab)(           )(A ^§b A§$ b)(A ^§b A§b b)=120hps__60pps__(A)(AA)=120pps
        NotNTSC:(Damaged?:)(Fix:)__(Pulldown?:)(ReversePulldown:)=48hps__24pps__(A)(A A..§..§...§..§..$)=120pps
       Damaged?:AaBcCdDdDd     ...'look' for picture #4=#5 (i.e. damaged pulldown)
            Fix:AaBa§cBcCdDd   ...'seen' - convert to normal 2-3-2-3 pulldown
      Pulldown?:AaBbBcCdDd     ...'look' for halfpic #3=#5 & halfpic #8=#10 (i.e. 2-3-2-3 pulldown)
ReversePulldown:AaBbCcDd       ...'seen' - reverse 2-3-2-3 pulldown to 48hps
Or, code-type formatting can make the notation more readable as follows:
30'fps[30pps__60hps__(AbCdEfGhIj)                              ...'look' for ersatz-NTSC
                     (                                         ...ersatz-NTSC 'firstseen'
                       (Ab)                                      ...'look' for 2 halfpics
                       (A A§b A§$ b)                             ...'firstseen' of 'firstseen'
                       (A ^§b A§$ b)                             ...'seen'
                       ()                                        ...'finalseen' is empty
                       =120hps__60pps__(A)(AA)=120pps
                     )
                     (                                         ...ersatz-NTSC 'seen' - apply decomb-x2-hr
                       (Ab)                                      ...'look' for 2 halfpics
                       (A ^§b A§$ b)                             ...'seen'
                       =120hps__60pps__(A)(AA)=120pps
                     )
                     (                                         ...ersatz-NTSC 'finalseen'
                       (Ab)                                      ...'look' for 2 halfpics
                       ()                                        ...'firstseen' is empty
                       (A ^§b A§$ b)                             ...'seen'
                       (A ^§b A§b b)                             ...'finalseen' of 'finalseen'
                       =120hps__60pps__(A)(AA)=120pps
                     )
                     (                                         ...ersatz-NTSC 'unseen'
                       (AaBcCdDdDd)(AaBa§cBcCdDd)__              ...convert damaged pulldowns to normal 2-3-2-3 pulldowns
                       (AaBbBcCdDd)(AaBbCcDd)=48hps__24pps__     ...apply reverse 2-3-2-3 pulldown
                       (A)(A A..§..§...§..§..$)=120pps           ...apply cinema-to-120fps interpolate-x16
                     )]120fps

Tip: At this point the reader should understand how strides work, how the notation enables nested substrides within strides (for example, (Ab) in the 3 'seen' terms that follow the (AbCdEfGhIj) 'look' term), and how judicious construction of 'look' terms obsolete many FFmpeg-style filters including 'split' 'select' and many more. In fact, the only way to implement multiple-segment, FFmpeg filter graphs is,
1, use FFprobe to dump a list of all the 'key_frame's, then
2, while playing the video, note the running times at segment changes, then
3, use those times and the 'key_frame' list to discover the frame numbers of each segment, then
4, construct a forest of 'split's & 'select's that use the segment frame numbers to route each segment to it's appropriate filter.
Unfortunately, the FFmpeg filter graph will be suitable for solely that single multiple-segment video. In contrast, Rehabilitation 3 is suitable for every multiple-segment video.

Author comment: The Rehabilitation repair & reformat notations above, attractive as they may be, are of little actual utility if there is no video tool that actually implements the notation.