Video Retargeting: Automating Pan and Scan

Sentence: 

In this paper we address the problem of adapting video for
viewing on smaller sized and differently aspect-ratioed displays than
they were originally intended by automatically adapting the video by
adaptively cropping and scaling the video.

Paragraph (beginning of the intro):

Viewing video on small screens is becoming increasingly common as
portable devices become more capable and popular. Unfortunately, most
of the source material was originally intended for larger displays,
such as televisions and theater screens. If such video is presented
na\"{\i}vely, by simply scaling it to fit the small screen, important
parts of the image may become too small to see. To make matters worse,
the small displays often have different aspect ratios than larger
ones, requiring either an anisotropic ``squish'' or padding the video
with blank space to fill the display in a ``letterboxed'' style. 

In this paper, we consider the problem of {\em video retargeting,}
that is, adapting a video so that it is better suited for viewing on a
display different than was originally intended. We present an {\em
automatic pan and scan} method that automatically determines how to
scale and crop the video based on an estimation of what is important
within the frame. The method attempts to maximize the amount of
salient content in the resulting video by either cropping shots to
remove less important parts of the image, or synthesizing
cinematically-plausible pans or cuts so that dynamic shots are fit
effectively on the small display. While our focus is on adapting
videos for small displays, the methods are also applicable for
automatically adapting wide format videos (such as feature films) to
other aspect ratios (such as standard television).

Outline:

- Intro
  - problem definition
  - need automation - since all kinds of different displays
  - necessity of information loss
	- downsampling (uniformly throws away details) vs. crop then
	  scale (totally throws away cropped stuff, also throws away 
	- we WILL throw away information - hopefully selectively
	- opportunity to tune to viewer's preference: preserve
	  cinematography vs. increase importance
  - pan and scan in television
	- not very popular since loses original presentation
	- TVs big enough losing some at top and bottom isn't big deal
	- letterboxing on tiny screen is bad
	- we zoom in addition to pan/cut
  - hard/impossible problem
	- can't know what's important (depends on story)
	- can't preserve everything
	- need to insure quality on each image, but continuity
	- composition / ``film arts'' big part of message /
	  communicates subtly
  - key ideas
	- work per-shot to make tractable
	- hueristics for whats important in a shot
	  (potential for non-automatic)
	- trade off preservation of overall image with maximizing
	  importance amount
	- introduce {\em cinematically plausible} virtual pans and
	  cuts
	  obey basic ``rules'' of film. directors violate these rules
	  for impact, but we don't want impact: we want to try to be
	  innocuous. 
? Overview Subsection (to give basic ideas/examples) - this way we can
  dive right in after the related work

- Related Work	
  (idea - put related work about salience in salience detection)
  - specific video adaptation
	- nothing really about pan and scan (not aware of automation)
	- not aware of automation: not suprising, only for feature
	  films going to TV, big enough market that deserves manual.
	- person understands story, content, ... can do as good a job
	  as possible <- at best a system could do as well
        - MSRA/Singapore/...
  - automatic image retargeting
	- why the problem is different	  
  - automatic composition
	- why its NOT what we do
  - using film knowledge
  - comparison with virtual videography
  
- Importance finding
  - why the problem is so hard in video (story, continuity)
    can't REALLY just look at one shot/not consider audio...
  - simplification: video salience + object recognition + hints
  - related work
    - static salience / importance
    - dynamic salience / importance
    - models for what we do
  - our video salience model

Idea: either have the ``optimization'' section, or put the material
scattered into the next sections

- Video retargeting as an optimization problem
  - work per shot (why this is bad, but necessary)
  - how we detect shots - emphasize could use better method, but ours
    seems good enough
  - basic objective function
  - other kinds of penalties to include

- Per-Shot Cropping
  (if we didn't do optimization, then include shots stuff)
  - why cropping is usually the best bet (why virtual pans are bad)
  - objective terms in choosing crops
  - algorithm
    - speedup tricks
  - examples

- Introducing pans for following
  - why its necessary or useful
  - constraints on pans for ``cinematic plausibility''
  - the optimization problem
  - the simplification for tractibility
  - examples
  - choosing when this is appropriate

- Introducing cuts (if we have it working)
  - why its necessary or useful
  - constraints on cuts for cinematic plausibility
  - the optimization problem
  - algorithm
  - examples
  - choosing when this is appropriate

- Coverage pans (if we can do *something*)
  - why it is necessary or useful
  - when it might apply
  - how to create one
  - how to decide when its appropriate

- Examples
  - performance ofthe implementation

- Evaluation
  - WE NEED SOMETHING HERE
  - want to say ``simplistic, but it works ok''

- Conclusion
  - retargeting is not just about size/aspect ratio
    other differences in small devices
	- dynamic range / color range
	- viewing condition
	- patience (want time compression in portable)
  - limitations on simplistic importance model
    - best bet hints: not automatic, but a lot less work than manual
      pan-and-scan, plus automate adapting to all the different sizes,
      user preferences
    - more information: less conservative about zooming in on what's
      really important
    - some automation possible: speaker (facial motion) detection,
      main character detection, ...
      still doesn't say anything about story
  - lack of a coverage model
    - important to see everything once - once its covered, less
      important to see it
    - video epitomes as solution (?)
  - film purists will never want to crop anything/alter
    cinematography. other fans of a particular actor might want to see
    zoom-in on that