MVtools... a few notes on creating 'super slo-mo' for free

Discussion in 'Camcorders, Action Cams & Video Making Forum' started by rogs, Sep 2, 2015.

  1. rogs

    rogs
    Well-known Member

    Joined:
    Mar 16, 2008
    Messages:
    2,386
    Products Owned:
    0
    Products Wanted:
    0
    Trophy Points:
    86
    Location:
    Dorset
    Ratings:
    +382
    I've posted a few notes online about creating good quality 'super slo-mo' clips using free software, which you might find helpful if you need to create some smooth 'ultra slo-mo clips', and are on a really tight budget.
    They were written for a 'non techie' person (my brother).. hence the step by step 'left click here'- 'right click there' type of approach. Hope that's not too distracting!

    You can find the notes here: MVTools Slo-mo notes

    I posted them following a conversation I had with my brother about very high frame rate slo-mo, and how the newer cameras - like the Sony RX-10 II - could now produce super quality results, with up to 2 seconds of 1000fps at full HD.
    (Any longer than 2 seconds at that frame rate and resolution - you still need something like a Phantom Flex of course!).

    I did suggest you could also get quite good results with software like 'Twixtor' , which is sold as a 'plug-in' for most NLEs. Not in the same league as original high speed footage of course, but way better than the faclities usually offered with most commercial editors - especially at very high frame rates....It's not cheap though!..
    I then mentioned that you can also get similar results for free.. using AviSynth and MVTools.
    But he was not impressed at how difficult he found it to set those up.. and learn how to use!
    So I wondered if a few simple notes might help?..... I've yet to find out how he's getting on with them!....
     
    • Useful Useful x 2
    • Like Like x 1
    • Thanks Thanks x 1
    • List
    Last edited: Sep 25, 2015
  2. dosdan

    dosdan
    Active Member

    Joined:
    Dec 17, 2014
    Messages:
    415
    Products Owned:
    0
    Products Wanted:
    0
    Trophy Points:
    31
    Location:
    Brisbane, Queensland, Australia
    Ratings:
    +47
    Thanks for the link, rogs. I managed to get MVTools2 up and running. It took a fair bit of mucking around. The error messages, when something goes wrong when attempting to load .avs script files in Vdub, aren't very helpful. Couldn't get AVSPmod operating. It showed an err msg upon starting and then closed before I could read it.

    Here are two examples of MVtools2. The first example uses the dropped ball sequence from Best Pro-sumer level camera / camcorder with slow motion capability?

    I've taken the 0.5x set from the HC-V750 (the 2nd example in the link). Here is the full 1920x1080 frame showing the cropped region used in the comparison:

    [​IMG]

    The 0.5x sequence looks OK because no interpolation is used:
    [​IMG]

    The 0.25x sequence with the interpolation done in the camera:
    [​IMG]


    The 0.25x sequence created from the 0.5x camera version by a further 0.5x reduction using MVtools2. This involves a number of extra steps and takes about 15x-30x longer than converting 0.5x-> 0.25x in-camera:
    [​IMG]


    Next, an MP4 11s 0.25x 60p uncropped sequence from the HC-V750 (so the interpolation has already been done in-camera) of a discus throw. I suggest you R.click the link and "Save As/Save Link As" first before playing:
    https://dl.dropbox.com/s/bzam9rqe60urm7u/slo-mo 0.25x.mp4

    And the same 0.25x sequence, but a further 0.5x reduction done with MVtools2. So this is now a 22s 0.125x 60p sequence:
    https://dl.dropbox.com/s/c6so3tfe45ezdgh/slo-mo 0.125x.mp4


    It would have been better to use 0.5x (in-camera) + 0.25x (MVtools2) to get to 0.125x, but I no longer had the 0.5x version.

    My MVtools2 0.5x .avs script:
    Code:
     source = AVISource("D:\Slo-mo\Discus slo-mo.avi")
    
    #####set parameters in the 'return Slowdn' line below to suit exact clip  slo-mo
    return Slowdn(source, 0, Framecount(source), .5, .5)
    
    ######The remainder of this script allows MVTools to control the slo-mo. --Thanks to 'lecapitan' from the Doom 9 forum for this clever piece of scripting....######
    
    function Slowdn(clip source, int startFrame, int endFrame, float startRatio, float endRatio)
    {  mfpsn = FramerateNumerator(source)
      mfpsd = FramerateDenominator(source)
      mclip = Trim(source, startFrame, endFrame)
    
      GScript(""" if (endFrame < startFrame) { temp = endFrame endFrame = startFrame startFrame = temp } """)
    
      super = MSuper(mclip, pel=4)
      backward_vec = MAnalyse(super, overlap=4, isb = true, blksize=8, truemotion=true, search=3)
      forward_vec = MAnalyse(super, overlap=4, isb = false, blksize=8, truemotion=true, search=3)
      slowdown = MFlowFPS(mclip, super, backward_vec, forward_vec, Round(mfpsn * 1.0/endRatio), mfpsd)
      slowdown = AssumeFPS(slowdown, mfpsn, mfpsd)
      startFrames = (mfpsn * 1.0/startRatio)/mfpsd
      endFrames = (mfpsn * 1.0/endRatio)/mfpsd
      framesRange = endFrames - startFrames
      currentFrame = 0
      counter = 0
      lastFrame = Framecount(slowdown)
      desiredFrames = 0
      removeFrameCounter = 0
      removeFrameThreshold = Float(endFrames)/Float(startFrames)
      overflow = 0
    
      GScript(""" while (counter < lastFrame) { removeFrameCounter = removeFrameCounter + 1
      if (removeFrameCounter >= Floor(removeFrameThreshold + overflow)) {
      if (overflow > 1) { overflow = overflow - 1.0 }
      else { overflow = overflow + removeFrameThreshold - Floor(removeFrameThreshold) }
      # Calculate the desired frames. You could use a higher power to make the transition even smoother.
      desiredFrames = startFrames + Pow(Float(counter)/Float(lastFrame), 4) * framesRange
      removeFrameThreshold = Float(endFrames)/Float(desiredFrames)
      if (removeFrameThreshold < 1.1) { removeFrameThreshold = 1.0 overflow = 0 }
      removeFrameCounter = 0  }
      if (removeFrameCounter == 0) { currentFrame = currentFrame + 1  }
      else { slowdown = DeleteFrame(slowdown, currentFrame) }
      counter = counter + 1 }  """)
    
      slowdown = DeleteFrame(slowdown, Framecount(slowdown))
      return slowdown}
    
    
    function Speedup(clip source, int startFrame, int endFrame, float startRatio, float endRatio)
    {  mfpsn = FramerateNumerator(source)
      mfpsd = FramerateDenominator(source)
      mclip = Trim(source, startFrame, endFrame)
    
      GScript(""" if (endFrame < startFrame) { temp = endFrame endFrame = startFrame startFrame = temp } """)
    
      super = MSuper(mclip, pel=4)
      backward_vec = MAnalyse(super, overlap=4, isb = true, blksize=8, truemotion=true, search=3)
      forward_vec = MAnalyse(super, overlap=4, isb = false, blksize=8, truemotion=true, search=3)
      slowdown = MFlowFPS(mclip, super, backward_vec, forward_vec, Round(mfpsn * 1.0/startRatio), mfpsd)
      slowdown = AssumeFPS(slowdown, mfpsn, mfpsd)
      startFrames = (mfpsn * 1.0/startRatio)/mfpsd
      endFrames = (mfpsn * 1.0/endRatio)/mfpsd
      framesRange = startFrames - endFrames
      currentFrame = Framecount(slowdown)
      counter = 0
      lastFrame = Framecount(slowdown)
      desiredFrames = 0
      removeFrameCounter = 0
      removeFrameThreshold = Float(startFrames)/Float(endFrames)
      overflow = 0
    
      GScript(""" while (counter < lastFrame) { removeFrameCounter = removeFrameCounter + 1
      if (removeFrameCounter >= Floor(removeFrameThreshold + overflow)) {
      if (overflow > 1) { overflow = overflow - 1.0 }
      else { overflow = overflow + removeFrameThreshold - Floor(removeFrameThreshold)}
      # Calculate the desired frames. You could use a higher root to make the transition even smoother.
      desiredFrames = endFrames + Pow(Float(counter)/Float(lastFrame), 4) * framesRange
      removeFrameThreshold = Float(startFrames)/Float(desiredFrames)
      if (removeFrameThreshold < 1.1) { removeFrameThreshold = 1.0
      overflow = 0}
      removeFrameCounter = 0 }
      if (removeFrameCounter == 0) { currentFrame = currentFrame - 1 }
      else {slowdown = DeleteFrame(slowdown, currentFrame)
      currentFrame = currentFrame - 1 }
      counter = counter + 1 } """)
    
      return slowdown  }
    

    Dan.
     
    Last edited: Nov 27, 2016
  3. rogs

    rogs
    Well-known Member

    Joined:
    Mar 16, 2008
    Messages:
    2,386
    Products Owned:
    0
    Products Wanted:
    0
    Trophy Points:
    86
    Location:
    Dorset
    Ratings:
    +382
    The 0.125x looks pretty good....very smooth!

    As you say, starting with the 0.5x would not have introduced any of the Panasonic interpolated 'new' frames, and would have just used 'new' MVTools frames. How much better - if at all - it would have looked, it's difficult to say?

    Although MVTools can produce excellent results, it's not perfect, and like it's commercial equivalents - Twixtor for example - certain types of motion can be difficult to 'predict' accurately.
    Legs 'crossing over' can be a problem for example. So you can get strange artifacts on occasion.

    For the task you have set yourself, you can actually use a much simpler script if you wish?
    The script I posted in the notes is really for timed slo-mo, so that you can select 'in and out' slo-mo points, 'ease in' and 'ease out' change rates etc....
    If you want to simply slow down a whole clip, the following script is probably easier to use:

    Code:
    AVISource()
    
    # This script will double the frame rate. .... To create a 0.5x clip..(i.e. replay at original rate)...
    
    #In Virtualdub go to 'Video -> Frame Rate -> Source rate adjustment -> Change frame rate to (fps)'  - Type in the original clip frame rate -> click OK
    
    super = MSuper(pel=2)
    
    backward_vec = MAnalyse(super, overlap=0, isb = true, truemotion=true, search=3)
    
    forward_vec = MAnalyse(super, overlap=0, isb = false, truemotion=true, search=3)
    
    MFlowFps(super, backward_vec, forward_vec, num=2*FramerateNumerator(last), \
       den=FramerateDenominator(last))
    
    
    
    To increase the frame rate further, you simply change the number '2' in the Framerate Numerator to a higher figure. (Remember you'll still need to enter the original clip frame rate value in Virtualdub)
    You can also alter some of the values, which can produce better results with certain clips.....the values of 'pel' in MSuper can be 1, 2 or 4.....As can the overlap values in MAnalyse.

    Shame you haven't had any luck with AvsPmod. It's really quite useful, because it can confirm the script validity, and give you much better error messages.
    As I mentioned, it doesn't need to be installed...I just downloaded it into its own folder, unzipped the files, opened the 'AvsPmod' folder and double left clicked the '.exe'.
    I then clicked on 'Associate .avs files with AvsPmod' from the Options menu..... I've had no problems with that here?...strange?..
     
  4. dosdan

    dosdan
    Active Member

    Joined:
    Dec 17, 2014
    Messages:
    415
    Products Owned:
    0
    Products Wanted:
    0
    Trophy Points:
    31
    Location:
    Brisbane, Queensland, Australia
    Ratings:
    +47
    While I like being able to go beyond a 0.25x slowdown, I'm not sure about the superiority of MVTools2 over Panasonic for a straight 0.25x version. Look at the interpolated frames in the 0.25x dropped balled example, and look at the details on the ball, rather than the artefacts around the ball. I reckon the surface details are clearer on the Panasonic version.

    I plan to repeat this dropped ball test. I'll get up on a ladder and turn the camera on its side (so the longer axis matches the direction of the motion), and then zoom in until the ball fills more of the frame when the ball is near the bottom of the drop, instead of framing the full extent of the drop as I originally did. In this new version the ball will travelling faster and the resolution should be better.

    Dan.
     
    Last edited: Sep 12, 2015
  5. dosdan

    dosdan
    Active Member

    Joined:
    Dec 17, 2014
    Messages:
    415
    Products Owned:
    0
    Products Wanted:
    0
    Trophy Points:
    31
    Location:
    Brisbane, Queensland, Australia
    Ratings:
    +47
    Found out the problem. I'd placed the AvsPmod directory under C:\Program Files (x86). But it seems that the AvsPmod program, when it goes to open, wants to create a new file immediately in write mode (last_session_.ses?). Win 10 won't allow it to do this so the program falls over immediately and the message window opens & closes much too fast to read.

    I got it working by placing the program directory on D: i.e. D:\AvsPmod.

    Rogs, thank you profusely for bringing this program to my attention. The much better error reporting of avs script problems (error type, script row & column) is most appreciated, after struggling to deal with Vdub's "unable to open file slo-mo.avs" err msg!!!

    It's been a long time since I've used Vdub & AviSynth (Win XP days). I vaguely seem to remember that running avs scripts from within the editor in VdubMod also used to show the errors this way too.

    Dan.
     
    Last edited: Apr 9, 2016
  6. rogs

    rogs
    Well-known Member

    Joined:
    Mar 16, 2008
    Messages:
    2,386
    Products Owned:
    0
    Products Wanted:
    0
    Trophy Points:
    86
    Location:
    Dorset
    Ratings:
    +382
    Glad you managed to sort out the AvsPmod problem. As you say, it makes editing and correcting Avisynths scripts a lot easier. I'm not running Windows 10 yet, so I still have a whole lot of adventures to look forward to there, I suspect.
    I don't think many folk on this board use programs like AviSynth and Vdub. Most seem to prefer the commercial editors..so we're probably having pretty much a 'one to one' conversation on this kind of issue. Each to their own, as they say...:)
     
  7. dosdan

    dosdan
    Active Member

    Joined:
    Dec 17, 2014
    Messages:
    415
    Products Owned:
    0
    Products Wanted:
    0
    Trophy Points:
    31
    Location:
    Brisbane, Queensland, Australia
    Ratings:
    +47
    I normally shoot 1080/60p 50Mbps MP4. At junior soccer this week I used 1080/60p 28Mbps AVCHD as that's what the camera was left in after some experimenting. I've chose Sports AE, so a fast shutter speed was utilised. The score was 6-1. I decided to include 1/4x slo-mo replays of each goal in my full match video.

    The quality of slo-mo would be better if shot in the V750's 1080/120p 28Mbps MP4 slo-mo setting. The camera packages these as 60p, rather than as 120p clips. So, when you play them back at 60p they take twice as long as the original event. Furthermore, it can do an in-camera conversion of these clips to 1/4x 60p, using interpolating. A 60s 1/2x clip (30s event) takes 120s to convert to 1/4x 60p 28Mbps MP4, which is relatively quick for a slo-mo conversion.

    The camera can shoot for a long time in slo-mo without overheating or other problems. I shot a 70 minute match fully in slo-mo, under poor lighting conditions. Since I knew the results would be so-so, rather then struggle with AF, I instead set the camera up to just cover our goal area and used fixed MF to pre-focus on the goal posts. There was no panning or zooming. The results are suitable for goal-keeping analysis purposes (all files should be downloaded first before attempting to play). Here is a great goal at 1/4x:

    https://dl.dropboxusercontent.com/s/0eur0xds3i6zkzl/Geat Goal 60p (resized) .mp4

    However, the problem with slo-mo shooting is that no sound is recorded. It would have been better if Panasonic had offered an option to make the files available as 1x 120p, with sound. Then you could slow them down in software, to 1/4x 30p, without sound, if required.

    So the way it is, you can't shoot a match in 1x and include slo-mo replays sourced from 120p. To get around this you would need to shoot in silent slo-mo and sync it with an externally recorded soundtrack. The only way to sync them for soccer is using a clapper board. I'll probably end up trying this, but I can see big sync problems if you pause/stop during a half, say because you pressed the Recording button again inadvertently, noticed it and quickly continued on. (You'd probably need to sync the new segment by using the clap-board again at the end.) In your editor you'd then process the 1/2x 60p -> 1x 60p, by dropping every second video frame, to speed it up so that the time-length of the video track matched that of the sound track. Theoretically, this should work fine.

    But what I'm presenting here today is slo-mo created in software from 1x 60p. Here is the 1x Full-HD 60p 28Mbps 16s clip of the lead-up to a goal. It was originally an AVCHD file, but I've repacked it (no video quality loss) as an MP4. You can try your own slo-mo conversion on it:

    https://dl.dropbox.com/s/3jkcdxs5nktb0lt/CLIP.mp4

    In Vegas I've slowed down the clip by using Ctrl-drag on the right-hand end of a clip. You can extend a clip's length out to 4x i.e. 1/4x. (You can also R.click on a video clip | Properties | change the playback rate to "0.25". But a 1x 4s clip would stay 4s long i.e. it's now only covering just 1s of the event. So then you need to L.click-drag the end of the clip until it's 16s long.)

    Vegas defaults to Smart Resampling when you change playback speeds. Considering 2 original frames with No Resampling, a 1/4x show-down will look like 1,1,1,1,2,2,2,2. So each frame is being repeated 3 extra times. Therefore, at 60fps, the effective change rate is 15fps. This makes the action slightly jerky.

    With Smart Resampling there is interpolation in the new frames inserted between 1 & 2. This interpolation is basic so it doesn't slow the program down much to create it, and tries to fool the brain into seeing a smoother transition. It looks strange if you single-step though it. A 5-frame sequence (both ball motion & camera panning, R->L):

    [​IMG]

    You can also notice the rolling-shutter distortion of the fencing upright during this fast pan. (BTW, the horiz. lines are power lines in the background.)

    Here is a comparison of this 16s sequence. (Trying to keep the download-size acceptable, I've dropped this comparison file to 720/60p, but the relative differences are still visible):

    1. 1x Original with sound (16s)
    2. 1/4x Vegas - No Resample (64s)
    3. 1/4x Vegas - Smart Resample (64s)
    4. 1/4x MVTools (64s)
    5. 1/8x Vegas - No Resample + 50% Velocity (127s)

    https://dl.dropbox.com/s/kslur5zr3amfl3p/Soccer Slo-mo - 2pass (2).mp4

    While playback-speed changes stop are 0.25, you can go much slower by using a velocity envelope on top of this: Insert | Video Envelopes | Event Velocity. I've included a 50% further speed reduction here as an example.

    Even with the jerkiness, the slo-mo clips are preferable because they give you extra display time to notice things in a fast situation.

    The MVTools shows obvious breakup, particular in the crossing-part of the sequence. This is an old program that hasn't been updated and, while I tried varying some parameters, I think it will have problems with fast motion in Full HD clips. The reason is, I believe, quoting from MVTools2 artifacts, but only at high resolution - Doom9's Forum

    The problem you're seeing is mostly due to a limitation in MVTools. For MFlow, MFlowInter, MFlowFPS and MFlowBlur the motion vector x & y values cannot exceed 127 or the results will be (increasingly) incorrect. It's because those functions use a single byte to hold the vector values rather than the 4-byte integer they are normally stored in. Using a Pel of 2 that's 127 half-pixels - faster motion will not interpolate properly in those functions. In fact MFlow is completely broken in the last "official" release, but cretindesalpes seems to have fixed that in the most recent dither-tools MVTools build (which you should be using). But the 127 limitation remains. It wouldn't be complex to fix, but it would require quite a lot of changes to those functions and a new resizer.
    Note: SVPFlow can handle motion vectors of up to 1023, so it's probably a better choice for fast-action slow-downs nowadays.

    Notice the slight vertical jerkiness in the Blue #6 player just after the goal is scored. This is better handled in the MVTools version compared to Vegas.

    Dan.
     
    Last edited: Nov 27, 2016
  8. Terfyn

    Terfyn
    Active Member

    Joined:
    Apr 19, 2013
    Messages:
    2,095
    Products Owned:
    1
    Products Wanted:
    0
    Trophy Points:
    66
    Location:
    North Wales UK
    Ratings:
    +241
    What sound would you expect it to record? The camera is in slow motion mode but the sound would still be in real time. It is not logical to expect the sound to be recorded because when played back it would be a low frequency roar. Panasonic got it right.
     
  9. dosdan

    dosdan
    Active Member

    Joined:
    Dec 17, 2014
    Messages:
    415
    Products Owned:
    0
    Products Wanted:
    0
    Trophy Points:
    31
    Location:
    Brisbane, Queensland, Australia
    Ratings:
    +47
    No they didn't. The slo-mo recoding is internally done at 120fps. So a 1s event is captured with 120 frames. But the metadata included in the saved data is adjusted by the camera to identify this clip as a 60p clip. At this playback rate the 120 frames take 2s to play i.e. a 1/2x slo-mo.

    If Panasonic had offered the additional option of 1080/120p, the 120 frames would have the sound included. If you don't want to do any slo-mo processing in PP, but just wanted the camera to do it for you, you would instead choose the 1/2x 60p saving option, without sound (as they currently offer as "slo-mo").

    But with 1080/120p, you get to record an action event with 8.3ms temporal granularity (10ms @ 100p). You can then slow this down in software to 1/2x or 1/4x This means from the one recording you can capture your action event, with sound, but later can also process parts of it as slo-mo action replays, without sound. (You just delete or mute the slowed-down audio for these parts.) Your final action event processed movie would be say 60p or 30p, (depending on how you rendered it), and combine mainly 1x stuff with sound, but with in-situ silent (or with added voice-over/music) 1/2x or 1/4x slo-mo replays of the significant bits.

    You can do this right now, as I'm doing, using a 1080/60p (normal) recording as the source. But doing so means than the temporal granularity in the source material is 16.6ms, so that later slow-downs in PP are more likely to be visibly jerky.

    In a 28Mbps 1080/120p recording, the audio part is only 1% of the total stream size, so the extra processing burden imposed on the camera by offering an option to include the sound, is minimal.

    So, as well as an option to do slo-mo, a 2nd option to do 1080/120p with sound would work just as well as a source for later slo-moing in PP, but this way you've also got the sound for a non-slo-mo version with sound.

    Dan.
     
    Last edited: Apr 19, 2016
  10. Terfyn

    Terfyn
    Active Member

    Joined:
    Apr 19, 2013
    Messages:
    2,095
    Products Owned:
    1
    Products Wanted:
    0
    Trophy Points:
    66
    Location:
    North Wales UK
    Ratings:
    +241
    You said it yourself. You would mute the sections of recording that is slowed. Panasonic have provided a simple slo-mo option. If you recorded sound at the slowed speed, it would not sound correct on playback. So why bother to record sound when any video editor will allow you to add sound from any source.
     
  11. dosdan

    dosdan
    Active Member

    Joined:
    Dec 17, 2014
    Messages:
    415
    Products Owned:
    0
    Products Wanted:
    0
    Trophy Points:
    31
    Location:
    Brisbane, Queensland, Australia
    Ratings:
    +47
    Here's part of the Vegas timeline where Goals #5 & #6 occurred, showing how I'm doing this with 1080/60p normal (i.e. not slo-mo) source material:

    [​IMG]

    1. As you can see, I've split a part containing the lead-up & the goal e.g. "5". (Splitting is done so I can copy just this section.)
    2. I then move the rest of the track to the right so I've got some blank timeline space after the goal.
    3. I copy the goal portion & paste it into the blank space.
    4. I then Ctrl-drag the right edge of the copy until the program will no longer stretch. This is at 0.25x playback speed. The copied clip is now 4x longer. During rendering, Vegas will replicate frames in thls section to produce the slow down. So the frame rate does not change. The fact that it has been stretched is indicated by the wavy horiz. lines.
    5. Next I ungroup the audio track from the video track in the copied portion and then delete the audio. (Alternatively I could just have muted this section, but the missing audio gives me a visual reminder in Vegas that this section is a slo-mo replay.)
    6. Then I bring the remainder of the track back to the left to remove the gap after the copied section.
    7. Finally I add fade-outs at the end of the 1x goal portion, at both ends of the slo-mo replay portion, and at the start of the following 1x normal part. These are the blue curves.
    Dan.
     
    Last edited: Nov 27, 2016
  12. rogs

    rogs
    Well-known Member

    Joined:
    Mar 16, 2008
    Messages:
    2,386
    Products Owned:
    0
    Products Wanted:
    0
    Trophy Points:
    86
    Location:
    Dorset
    Ratings:
    +382
    (BTW - Your first link has a '/' missing Dan )

    The second clip shows pretty much what you'd expect. The 'no resample' Vegas option simply repeats frames.....in the case of clip 5 it shows each frame 8 times. For 'forensics' analysis of the football skills, that's probably fine.
    I'm not impressed with the Vegas 'smart re-sampling'. The interpolated frames seem to just introduce 'ghosting' of both the before and after frame. I think I prefer the frame repeat 'jerkiness'!

    The Mvtools clip is the smoothest, but it does of course have the problems that all optical flow processing can introduce. In the case of the fast movement - like the cross - there is simply too little information to make any serious sense of some the motion vectors, and some of the constructed frames have some weird artifacts The ball becomes a bit of a mess in that part of the clip.

    In your clip, MVTools does make a pretty good job of the other aspect that can cause problems for optical flow. That of objects 'crossing over' (like legs!). Because it is not very intelligent, the software sees an overlaid object in 2 dimensions, and has to 'guess' which way each bit is going next. That can cause some weird effects... not too bad here though.

    I think there are two ways of looking at this. If you're doing slo mo to 'show off' a specific part of a clip, then you probably wouldn't slow the whole thing down, and MV tools can do timed slomo - with ease in and outs etc - pretty well.
    If a sports clip is being used for analysis, then the frame repeating is probably best. A bit jerky of course, but no strange new artifacts introduced at all.

    As you say, MVTools is now pretty old, and some of the newer versions of optical flow tools may show improvements. But none are going to be perfect.... you can't change the laws of physics! The best way of doing high quality slomo is still to use a Phantom Flex or similar (not likely to happen for us hobbyists though!)
    Where optical flow really works is where it can highlight a special moment in a clip, and the end result can be improved 'artistically' by controlling the timings of the 'special' event, including ease in and out timings...

    If I'm using audio with a slo-mo clip, I simply edit and rebuild the audio track to suit. I always work from intraframe versions of my clips, and that process creates a linear pcm audio track. Much easier to work with separately in an audio editor, and then add the reconstructed track, before rendering the final edit.

    One thing your clip does do - It reminds me why I hate rolling shutters! :)
    Although as the whole world in now pretty much used to them these days, the slanting verticals and 'jello' wobbles don't seem to worry most folk. (no 'jello' here though)
    It's a bit like when mp3 audio arrived..... It wasn't as good quality as the CD tracks it replaced, but it was more convenient, and most folk didn't really care much about any quality loss.
    I do think it's a shame that some broadcasters now find it acceptable to use rolling shutter cameras for fast moving sports items though. That can look very amateurish, compared to the 'old days'...
     
    Last edited: Apr 19, 2016
  13. dosdan

    dosdan
    Active Member

    Joined:
    Dec 17, 2014
    Messages:
    415
    Products Owned:
    0
    Products Wanted:
    0
    Trophy Points:
    31
    Location:
    Brisbane, Queensland, Australia
    Ratings:
    +47
    Thanks for that. Corrected. The link URL was actually OK - it was the text associated with it (the link title, in this case, the "apparent" URL) that had the typo.

    Dan.
     
  14. rogs

    rogs
    Well-known Member

    Joined:
    Mar 16, 2008
    Messages:
    2,386
    Products Owned:
    0
    Products Wanted:
    0
    Trophy Points:
    86
    Location:
    Dorset
    Ratings:
    +382


    Motion vector limitation fixed in latest version - see comment from Fizick here:

    MVTools without idx - Page 13 - Doom9's Forum


    Download here:
    MVTools

     
  15. dosdan

    dosdan
    Active Member

    Joined:
    Dec 17, 2014
    Messages:
    415
    Products Owned:
    0
    Products Wanted:
    0
    Trophy Points:
    31
    Location:
    Brisbane, Queensland, Australia
    Ratings:
    +47
    Rogs, thanks for the info about an update. I'll try this version out.

    I've received a lot of favourable comment from coaches & parents since I included 1/4x slo-mo replays of goals in my last match video. So I've spent some time evaluating slo-mo via the camera (V750) and via VirtualDub and/or in Vegas (no resampling). The sequence of 0.25x comparisons to be presented here are:

    60p project output (the output frame rate is the same as that of the source)
    1. 1x from the camera slowed to 0.25x in Vegas by the replication of frames i.e. Frames 1,2,... becomes 1111,2222,...
    2. 0.5x from the camera slowed another 0.5x in Vegas i.e. 1,2 -> 11,22
    3. 0.5x from the camera slowed another 0.5x in the camera with interpolation

    https://dl.dropbox.com/s/wczh1otor6ekd86/60p_slomo_comparison.mp4

    Since the Internet distribution format of this 1080p60 source material is usually 480p30 on YouTube, changing the frame rate outside Vegas to 30p and then setting the Vegas project rate to 30p will itself produce a 0.5x slow down from the original 60p source material:

    30p project output (the output frame rate is lower than that of the source)
    1. 1x 60p slowed to 0.5x 30p in VDub and a further 0.5x in Vegas
    2. 0.5x 60p slowed to 0.25x 30p in VDub
    3. 0.25x 60p slower to 0.125x 30p in Vdub

    In the last example, I've included a 1/8x slow-down.

    https://dl.dropbox.com/s/5a7kjk29pd54f66/30p_slomo_comparison.mp4

    You can see that accepting a 30p project output frame rate will result in less frame duplication in 30p #1 i.e. 11,22 instead of the 1111,2222 needed for 60p project output. So it should look a bit smoother.

    In VDub, to relabel 60p material as 30p, I used Video | Frame Rate | Source rate adjustment - Change frame rate to 29.97. (30fps is only a convenient approximation.) There is no change in the number of frames, so 60 frames, which would play for 1s @60fps, will now play for 2s @30fps.

    In VDub the data will be decompressed during processing. You can save the processed clip in an uncompressed form, but video data is enormous. For example, in Full HD 60p, each minute requires:

    frame height x frame width x 3 bytes per pixel [a byte each for R,G,B] x frame-rate x 60s

    = 1920 x 1080 x 3 x 60 x 60 = 22,394,880,000 byes/min = 22.4 GB/min

    The bit rate per second is:

    1920 x 1080 x 24 [24-bit colour] x 60 frames = 2,986 Mbps or approx. 3 Gbps.

    The HC-V750 creates a new Full HD 60p 50Mbps MP4 every 11 mins. So to cover a 30-45 min. soccer half (duration varies depending on age group) you're going to consume a frightening amount of disk-space unless you re-compress the raw data. Now the vast majority of video compression used is lossy, so there is a quality loss each time you re-compress. The solution is to use a fast, low-compression "intermediate" format which suffers little additional quality loss and can endure multiple re-compression cycles without much obvious degradation.

    Also high video compression requires spatio-temporal compression i.e. compression both inside each frame (like JPEG in a stills shot), and between frames (compression across time that looks for similarities between frames). In a static shot much of the background will be unchanged, while with a panning shot a lot of the pictures elements will be the same, just displaced from one frame to the next. So splitting the image into small blocks, and using motion vectors to describe the direction & displacement that each block has moved from its position in the previous frame, is a very effective form of data compression.

    I'm using the free Canopus HQX codec as an intermediate compression format, packaged by Vdub into an AVI file that will be used in Vegas. In an intermediate format, only spatial compression is usually used so each frame is self-contained and self-sufficient i.e. a "snapshot". This makes editing much easier than with spatio-temporal compression where earlier & later frames can contain information needed to correctly reconstruct the current frame.

    Here is a table showing file-sizes for 1 min of Full HD 60p.

    Column 1 Column 2 Column 3 Column 4 Column 5 Column 6
    0 Format Avg/Max Video Bitrate (Mbps) Bits/(Pixel*Frame) Size (GB/min) Size (%) Comp. Ratio
    1 854x480p30 MP4 Internet distribution(small) 3.0/4.0 0.248 0.024 0.11% 937:1
    2 1680x944p60 MP4 8GB USBstick distribution (large) 13.9/26.0 0.146 0.105 0.47% 213:1
    3 1920x1080p60 MP4 Videocamera ("28M") 24.5/26.0 0.196 0.183 0.82% 122:1
    4 1920x1080p60 MP4 Videocamera ("50M") 45.0/49.2 0.362 0.338 1.51% 66:1
    5 1920x1080p60 HQX Std. 132 1.065 0.993 4.44% 22.5:1
    6 1920x1080p60 HQX Fine 169 1.359 1.27 5.66% 17.6:1
    7 1920x1080p60 HQX Superfine 478 3.843 3.59 16.0% 6.25:1
    8 1920x1080p60 Uncompressed 2,986 24 22.4 100% 1:1


    Note: "1680x944p60 MP4 8GB USBstick distribution (large)" is chosen to fit a 70 minute junior match @ 14Mbps on to a 8GB USB Stick.

    With digital video, when comparing formats, Bits/(Pixel*Frame) is a meaningful way of seeing how many bits are being used to encode the value of a pixel in a frame. The higher this BPF value, the less usually-lossy compression is being applied, and usually the better the potential quality.

    You've probably noticed in the two comparison clips above that the 1x portion looks sharper and closer. When you switch to Slo-mo mode, the camera appears to zoom in:

    [​IMG]



    This means that, although the camera's output size & format is the same (Full HD 60p) for both Normal & Slo-mo, a smaller portion of the sensor is being used in Slo-mo, compared to Normal mode. This smaller area is then being resized up to Full HD, hence the "magnification" effect.

    The change in linear dimension is 1.37x. Squaring this to get the area, Slo-mo uses 53% of the pixels of Normal mode. So the resolution suffers. Here is the closer comparison of part of these two images:

    [​IMG]

    So while the slo-mo is marketed as "Full HD", this is upsized and of lower quality then if true Full HD was being used. It would be less appealing to market a camera that offers 1400x788 60p slo-mo. This is also the likely reason why the camera does not offer a 1x "Full HD 120p" mode (with sound): either the sensor readout speed or the processing electronics can't operate this quickly.

    This means my plan to:
    1. Shoot a match in slo-mo
    2. Change the frame rate afterwards in VDub so the same clip could also be used for 1x
    3. Sync it with a soundtrack from an ext. audio recorder
    4. Use it in Vegas

    so I could have both higher quality slo-mo, as well as normal speed-with-sound, from the same video, (all of which I've successfully done on a 45min test shoot here in the house), is doomed to failure because the sped-up slo-mo, used to produce the normal-speed footage, will be inferior to a normal-speed Full HD recording due to the slo-mo originating from "faked" Full HD.

    Dan.
     
    Last edited: Nov 27, 2016
  16. sound idea

    sound idea
    Active Member

    Joined:
    May 2, 2009
    Messages:
    121
    Products Owned:
    2
    Products Wanted:
    0
    Trophy Points:
    21
    Location:
    Rickmansworth
    Ratings:
    +14
    Thanks rogs and dosdan for a very interesting thread. Makes me want to try these techniques!
     
  17. rogs

    rogs
    Well-known Member

    Joined:
    Mar 16, 2008
    Messages:
    2,386
    Products Owned:
    0
    Products Wanted:
    0
    Trophy Points:
    86
    Location:
    Dorset
    Ratings:
    +382
    Give it a try!... my notes HERE were originally written for someone without a 'techy' background - my own brother - hence the 'left click here' - 'right click there' approach ..
    ..So hopefully are they are not too complex, if taken step by step. Once you've done it once, it should then get a lot simpler to do again..... Mostly just 'copy and paste' for the script files!

    If you do give it go, let us know how you get on.......
     
  18. sound idea

    sound idea
    Active Member

    Joined:
    May 2, 2009
    Messages:
    121
    Products Owned:
    2
    Products Wanted:
    0
    Trophy Points:
    21
    Location:
    Rickmansworth
    Ratings:
    +14
    I certainly shall but please don't hold your breath .....
     
  19. dosdan

    dosdan
    Active Member

    Joined:
    Dec 17, 2014
    Messages:
    415
    Products Owned:
    0
    Products Wanted:
    0
    Trophy Points:
    31
    Location:
    Brisbane, Queensland, Australia
    Ratings:
    +47
    I've noticed something about the processing priority in Vegas. I've been doing a number of 0.25x slo-mo replays, created by by Ctrl-dragging the right edge of a region, of goals, saves, trips, fouls & other significant soccer incidents,

    I've also started to do more Pan/Crops, usually in a replay, so that the area of interest can be better examined.

    A problem arises if you do both slo-moing & pan/cropping on the same segment. A visible horizontal jitter appears on sections where the camera has done a fast pan. This appears to be a processing sequence issue where the frame replication implicit in a Vegas speed-change occurs before the pan/cropping. If I frame-by-frame through a jittery part, I can see 3 frames moving in one horiz. direction and the 4th frame appears to jump back a bit in the opposite direction.

    What needs to be done to prevent this is to perform the speed-changing replication after the pan/cropping. While Vegas allows you to reorder the processing of plugins, including Pan/Crop, you have no control over when the speed-change occurs in the processing.

    The way around this is to:
    1. Pan/crop the relevant portion.
    2. Render this portion to an intermediate video format .AVI.
    3. Import this new clip.
    4. Then Ctrl-drag the new clip to slow it down. (You need to switch off smart re-sampling each time.)
    5. Since pan/cropping means you are reducing the resolution, I find it worthwhile to apply some sharpening to the stretched clip.

    With the free Canopus HQX intermediate format, you can re-render multiple times without a noticeable impact. I'm created a rendering template called "Full HD 1080-60p HQX (no audio)" which I use for this purpose.

    Here is an example (download first, before playing): https://dl.dropbox.com/s/amvplz8ll3vvi79/Pan and Scan Slo-mo ordering example.mp4


    While Vegas only offers a 0.25x slo-down, you can slow-down, save, import, slo-down again, re-processing multiple times. For example, if you do this twice, e.g. Save1a.avi & Save1b.avi, you slow down to a 1/16th speed. The action, with each original frame now occurring 16 times, doesn't look smooth, but it becomes like viewing a sequence of stills. This is handy to position this very slow sequence into the critical incident part of a 0.25x sequence when the action is very fast and you want a bit more time to linger, examining hand or foot positions.

    Dan.
     
    Last edited: Nov 27, 2016
  20. rogs

    rogs
    Well-known Member

    Joined:
    Mar 16, 2008
    Messages:
    2,386
    Products Owned:
    0
    Products Wanted:
    0
    Trophy Points:
    86
    Location:
    Dorset
    Ratings:
    +382
    That sounds even more long winded than applying an MVTools script! :)..... Converting to an intermediate like HQX gives you an intraframe file that's simple to scrub and determine exactly where you want your 'slo-mo' to happen.
    It's even easy to slow down and speed up gently as you approach your critical frames.

    And of course the result is smooth as well.

    The description and sample script on this page of my notes shows what I mean in detail:

    SLO-MO Edit

    As we've mentioned before, it's often a good idea to only slow down specific parts of a clip, and starting with an intraframe file makes it that much easier to edit the script with frame accuracy, and create the slo-mo exactly where you want it.
     
  21. dosdan

    dosdan
    Active Member

    Joined:
    Dec 17, 2014
    Messages:
    415
    Products Owned:
    0
    Products Wanted:
    0
    Trophy Points:
    31
    Location:
    Brisbane, Queensland, Australia
    Ratings:
    +47
    I'm working on improvements for next season's soccer videos. I plan to try MVTools again for 0.25x slo-mo replays. Currently I'm using Vegas' speed change which, for a 0.25x slow-down, replicates each frame 3 times i.e. frame 1,2,3 -> 1,1,1,1,2,2,2,2,3,3,3,3. I'm hoping that the frame interpolation in MVTools will look better than straight frame replication. (Nobody I know likes the result of Vegas' Smart Resample frame interpolation.)

    I'll be dealing with lots of short clips of interesting incidents e.g. G1, G2, G3 (goals), F1, F2 (fouls), S1, S2, S3 (saves), C1, C2 (collisions) etc. These are exported from Vegas by being rendered as HQX-with-no-audio AVIs. Then they will be opened in an AVS (Avisynth script file) which, in turn, was loaded by VirtualDub running a VCF (Vdub configuration file) to save the slo-moed version as a new (longer) HQXed AVI. I'll then re-import the processed clip back into Vegas.

    The problem with AVS media-file loading is that the script would need to be altered each time, as the clip names are all different, which is a pain when there are lots of clips. What I want to do is to drag-and-drop each new 1x clip onto a batch-file icon which then passes it to Vdub in the AVS. To do this, I need to set an environmental variable (EV) and then have the AVS read that EV. But Avisynth can't do this.

    Suspecting that there might be some plugin out there that added this capability, I asked in the Doom9 forum. It turns out that the RT_Stats plugin can add the reading of an EV, as well as a zillion other things, to Avisynth.

    RT_Stats, Compile-time/Runtime Functions v1.43 - 08 Oct 2014 - Doom9's Forum

    So what I'm developing is a system where I can drop each 1x AVI clip as it's rendered, e.g. G1.avi, on to Slo-mo4.bat. This will set the clip name as an EV and then invoke Vdub, telling it to load slo-mo.AVS & HQXoutput.VCF & to close Vdub when finished, having produced a 0.25x version of the clip. The AVS file will be able to read the EV containing the clip name and will open this and slo-mo it.

    In the development phase, I'm temporarily hard-wiring the clip name into the batch file so I don't have to keep either specifying it on the command line or drag-and-dropping it. I'm also using AVSPmod instead of Vdub to load the AVS file. Here is a demonstration of using an EV to pass a clip name into an AVS script:

    EVtest.bat
    Code:
    @echo off
    set video=D:\Slo-mo\CLIP.mp4
    D:\AvsPmod\AvsPmod.exe D:\Slo-mo\FromBatchfile.avs
    
    Basically, if you have the auto-loading of Avisynth plugins set up, you can demonstrate it with 1 line in your AVS. (I'm using video as the name of the EV containing the clip name):
    Code:
    FFVideoSource(RT_GetSystemEnv("video"))
    
    Here is the AVSPmod screen, which has been opened by the batch file, after I press the "Play" button to start running the AVS script.

    [​IMG]

    Dan.
     
    Last edited: Nov 30, 2016
  22. rogs

    rogs
    Well-known Member

    Joined:
    Mar 16, 2008
    Messages:
    2,386
    Products Owned:
    0
    Products Wanted:
    0
    Trophy Points:
    86
    Location:
    Dorset
    Ratings:
    +382
    I don't know whether using Macros from within AvsPmod would help with your intended work flow at all?... Macros

    RecentlyI have been exporting more directly from AvsPmod using the 'Tools - Script encoder (VFW)' to create my HQX files directly from within AvsPmod.....

    ...I find it can be a bit quicker than running every little script change through Vdub again... and avoids the double colour space conversion that running through Vdub's 'Full processing mode' entails....
     
  23. 200p

    200p
    Active Member

    Joined:
    Oct 13, 2008
    Messages:
    1,119
    Products Owned:
    0
    Products Wanted:
    0
    Trophy Points:
    66
    Ratings:
    +240
    The "1000 fps" on the RX10 M2 isn't really full HD though - it's upscaled to 1080p from a much lower resolution when at that frame rate.

    I haven't looked through this properly yet but surely the same tools could be used for frame interpolation (ie. converting 24p/25p to 30 fps or higher). Also Twixtor/optical flow can produce noticeable warping. Couldn't neural networks be made to potentially produce better results than Twixtor/optical flow etc.
     
  24. rogs

    rogs
    Well-known Member

    Joined:
    Mar 16, 2008
    Messages:
    2,386
    Products Owned:
    0
    Products Wanted:
    0
    Trophy Points:
    86
    Location:
    Dorset
    Ratings:
    +382
    Yes, I have seen other comments about the 'upscaled slomo'. Only to be expected I suppose, when you see the price of Phantom Flex professional slo-mo cameras!

    Optical flow techniques - whether it's payware like Twixtor, or freeware like MVTools - can be very good - certainly better than the usual frame doubling or simple interpolation of most editing software. But it does have limitations, and you can get some very odd artifacts. ..
    Optical flow software often doesn't know quite what to do with things like legs crossing over, for example.... It simply can't work out which way which leg is supposed to be moving in! ... and there are of course other similar situations that produce some weird results.
    OK if you've experimented for free, but a bit disappointing if you've coughed up some serious cash for Twixtor, I would think?...:)

    MVTools Optical flow can be useful for frame rate conversion. See this post for example:
    Question - Any PAL frame rate smartphone cameras yet? The samples there worked quite well.... Certainly better than the usual 'jittery' frame dropping (or duplication).

    Maybe neural networks are the 'next generation' for this work?.. although it's difficult to see how you can work out directions of movement with only 2D sources to work with.....
    Now with 3D sources.....who knows?...
     
    Last edited: Dec 3, 2016
  25. 200p

    200p
    Active Member

    Joined:
    Oct 13, 2008
    Messages:
    1,119
    Products Owned:
    0
    Products Wanted:
    0
    Trophy Points:
    66
    Ratings:
    +240
    I haven't got the paid-for Twixtor though - I've just used the demo version (that overlays stuff onto the encoded footage). Have also used After Effects' own "pixel motion" interpolator which can also warp - probably more than Twixtor (though with Twixtor you can apparently manually do stuff to make the interpolation better, like adding various mattes etc.).
    I haven't looked properly into it but I imagine you could, eg. take 2 frames (or more) at a time and output 1 frame. eg. get 50p video, and make it learn the in-between frames (eg. basically to convert 25p to 50p). eg. from the "25p" video, get two frames (that you want to create the inbetween frame for), for each of those frames maybe create a neuron for every pixel in them (actually more like 3 neurons - for every pixel (for R,G,B). Also, I imagine you wouldn't have to create neurons for every single pixel in a frame - maybe a much smaller window?). Have whatever number of hidden layers are needed (I've no idea how many), and have output neurons for all the pixels in the output frame.

    Instead of just doing it based on 2 source frames to interpolate I assume it would probably give better results if say 4 frames were used for input neurons (and you could still have one frame worth of output neurons I'd think).

    [Note: I'm assuming lots above about the neural network - I don't really know if it would/could work like that]

    It would be interesting if there was stats on which is the best (most accurate) interpolator (ie. for encoding) - especially where no manual tweaking is necessary (eg. just default settings in things like Twixtor/other programs/codecs) - and stats on how close each of the available interpolators are to a reference video shot at real higher rate (eg. at 60/100/120 fps etc.).
     
    Last edited: Dec 4, 2016
  26. rogs

    rogs
    Well-known Member

    Joined:
    Mar 16, 2008
    Messages:
    2,386
    Products Owned:
    0
    Products Wanted:
    0
    Trophy Points:
    86
    Location:
    Dorset
    Ratings:
    +382
    Some of the notes here: MVTools - especially the sections on MSuper and MAnalyse - make quite interesting reading (although much of it goes over my head a bit!).
    Certainly goes into quite a lot of detail how the various aspects of MVTools2 approach the various problems with different aspects of interpolation.... and there are quite lot of settings to 'tweak' with MVTools2 - some of which can make quite a difference when applied to individual clips...
     

Share This Page

Loading...
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice