Automated 24p pulldown workflow using DGIndex and VirtualDubMod
What is this application?

There has been a lot of discussion on some of the video forums regarding the 24p pulldown removal issues with the Canon HV20. Since the HV20 doesn't put flags on the frames, most NLE's are not able to properly remove the pulldown. Farnsworth and Mike Dulay were able to put together some batch files and perl scripts to help automate some of the manual workflow using free tools such as DGIndex and VirtualDub to remove the pulldown. This process seems to really do a good job for free tools but there were still some manual steps involved. This application ties DGIndex & VirtualDub together to fully automate the workflow.

What You Need First:
Mike has put together a great blog which will tell you about installing the freeware tools and the basics of this workflow. Before using my app, make sure you follow his setup guide for AVI Synth, Virtual Dub and DGIndex. You can find that here: http://yousillyman.blogspot.com/

Here are the basic steps to get this working:

  1. My application was written in C#, so you will need the Microsoft.Net 2.0 Framework installed on your machine.
  2. Install AVI Synth Application
  3. Install DGIndex Application (Doesn't matter where. example C:\Tools\DGIndex\ )
  4. Install VirtualDubMod Application  (Doesn't matter where: example C:\Tools\VirtualDubMod\ )
  5. TIVTC v1.01 plugin (Copy the DLL to your AVI Synth plugins folder)
  6. MPASource Plugin (Copy the DLL to your AVI Synth plugins folder)
  7. Install HV20Pulldown Application (my stuff) Open the zip file and extract the files to you machine. Where doesn't matter. For example, save it to: C:\Tools\HV20Pulldown\ etc.. The zip contains the .exe and a copy of the Template.AVS file.
  8. Create a "working directory" for the automated workflow. For example: "C:\Tools\Working\". Make sure the directory contains enough free space. This is where the final output file will be created, as well as all of the files created by DGIndex etc.
  9. From the location where you installed DGIndex, copy the DGDecode.DLL file to the AVI Synth plugins folder.

Otional Installation Components:

The below codecs are optional, but produce good quality video with reasonable file sizes.

  1. Lagarith Codec
  2. Huffyuv Codec
Running the automated workflow application:

Now that you have the tools installed, the next instructions will help you start processing your 24p clips.
  1. Launch he HV20Pulldown.exe application from the directory you saved it to.
  2. The application now needs to know where the other tools are located, as well as the working directory you created. Click the buttons next to each of the paths, which will open a file finder dialog. Fill each of these in for: Working directory, DGIndex.exe application, VirtualDubMod.exe application and the AVS template file.
  3. Using the checkbox, specify whether you want to have the application automatically fix the required audio delay (recommended).
  4. Specify the compression template that you wish to use. This actually modifies the .syl script that will be passed to VitrualDub. I have created a few sample ones, but if you want to, you can create your own using the "custom" option in the drop down list. See the bottom of this page for more information about creating your own custom scripts. The default compression template is for an uncompressed AVI. This works great, but will take a very long time and will generate very, very large files.
  5. Click on the "Video Files" tab page. You can use the Add or Remove buttons to add video files (or remove them) that you wish to process. The original files you specify will not be modified in any way and will be left in their original locations. Specify as many files as you want, we'll process them all.
  6. When you are ready to start processing, use the "Processing -> Run" menu option. If this menu option is greyed out, check the status bar at the bottom of the page. The status bar will let you know what you are missing, or have incorrectly setup. Fix those issues and try running again. You can also use the F5 shortcut to run.

That's pretty much all it takes. After you've set all of your app locations and working directory, the application (when closing) will save these settings so that you do not have to keep updating them in the future. There is also a "Open" and "Save" menu option that you can use to save off custom templates. This is especially useful if you create custom VirtualDub script jobs and don't want to keep re-typing them.

You can see some sample screen shots listed below as well as some other information about the application and creating your own VirtualDub scripts.

Notes:

Here are links to some threads where a bunch of people have been discussing this issue...
http://www.dvinfo.net/conf/showthread.php?t=92893 and  http://hv20.com/showthread.php?t=218


In my application, you will define a "working directory" where the temporary files and final output file will be placed. You will also provide the paths to DGIndex, VirtualDub (preferably VirtualDubMod) and your AVI Synth template file. You can specify the codec that VirtualDub will use as well. I put some default settings up there but that doesn't mean you have those codecs. For sure you have the Uncompressed AVI but beyond that I will make no guarantees. You can also create a custom one yourself. For more information on that, see the very bottom of this page.

Once you've defined those settings, you then specify the list of source video files that you want to convert. After that, just click the run button and watch as DGIndex & VirtualDub do their business. When it's all over, the working directory will have your output AVI files ready for you to use. You can delete all the other temporary files that are created by DGIndex. You won't need those anymore.

Source Code:
I will be putting the source code up for anybody who wants it later this week. I just need to finish doing some commenting in the code first.

Help and Information
Here are a couple of screen caps of the application. Below the images is a list of the different options that are available and what you can do.

General Tab
Working Directory Specifies the directory that will be used by the script to copy the video files (from your timeline). This is the directory where all work will be performed and the output files from DGIndex and Virtual Dub will be placed.
DG Index Path Specifies the full path and file name to the DGIndex.exe application on your machine.
Virtual Dub Path Specifies the full path and file name to the VirtulDubMod.exe application on your machine. I have tested this script only with VirtualDubMod and not VirtualDub (regular).
AVS Template Specifies the full path and file name of the AVI Synth Template file which will be used by DGIndex and VirtualDub.
Calculate Audio Delay Specifies whether the script will parse the AVS file created by DGIndex to determine any required audio delay.
Compression Specifies the compression (or really the whole job script) for VirtualDub. You can choose from one of the "presets" or create your own custom job script. See notes at the bottom of the page.
Edit ... Available only when using a custom compression. This allows you to enter the script information that will be used by VirtualDub during processing. See the bottom of the page.
Preview ... The script definitions for Virtual Dub are stripped of the calls for .Open, .subset, .Close and .SaveAVI. These are added by my script automatically. Preview lets you see the script more or less how it's passed to Virtual Dub.
Add ... Opens the open file dialog and allows you to add video files that you intend to process.
Remove Deletes all currently selected video files from the list of video files to be processed.
Run Executes the script workflow. If this button is disabled, check the status bar at the bottom of the page to see what setting has a problem.
Processing Menu: Clean WARNING: Clean will delete ALL files in the working directory. If you do this and delete things you didn't want to delete, it's not my fault. I warned you!
File Menu: Load ... Loads a script template that will update the General Tab with information that you previously saved using the Save ... button.
File Menu: Save ... Saves all of the information on the General Tab to an XML file which you can use to later load and execute. This does not apply to the video files tab.

The Script Editor

The script editor allows you to modify the default compressor templates or create your own new modified template that you can reuse. To modify an existing template, select it from the drop down list at the top of the page. Next, modify the contents of the script in the text area. When you are satisfied with your changes, click the "Save" button next to the compressor drop down.

To create a new compressor template, simply type in a new name for the compressor in the drop down list. Next add or modify the script text in the editor window. When you are complete, click the "Save" button.

You can also remove templates you no longer need by clicking the red "x" button next to the save button.

The application will always append the calls for VirtualDub.Open, .subset, .SaveAVI and .Close. Those are shown above and below the edit area of the dialog. You should NOT include those calls yourself. You should only provide the information required between those calls.

General Tab 

Working With VirtualDub Job Scripts:

Here is some information about the VirtualDub Script API http://www.virtualdub.org/docs/vdscript.txt. It is a good resource to use. Some caveats about these .SYL files and how you need to create them in my script editor. Firstly, I automatically will add the calls for .Open, .subset, .Close and .SaveAVI. If you create a custom script, DO NOT put those calls in there. Only give me the script information between those calls. below is an example SYL file generated by my script. In this case, the first line and last 4 lines are what is added by my script. In a custom script, you would add everything between those lines. You can use the VirtualDub UI to generate this file as well, just setup your compression methods to suit your tastes and the click File->Save As but instead of saving, tell it to create a job instead. Then you can look at the Job list and export that to disk, where you can grab the content required to give to my script. Ideally, I could pop up those same dialogs, but that would take more time than I have right now for this. Maybe in a later release I'll allow for that though... VirtualDub.Open("C:\\Tools\\Working\\Short0001_.avs", "", 0); VirtualDub.RemoveInputStreams(); VirtualDub.stream[0].SetSource(0x73647561,0); VirtualDub.stream[0].DeleteComments(1); VirtualDub.stream[0].AdjustChapters(1); VirtualDub.stream[0].SetMode(0); VirtualDub.stream[0].SetInterleave(1,500,1,0,0); VirtualDub.stream[0].SetClipMode(1,1); VirtualDub.stream[0].SetConversion(0,0,0,0,0); VirtualDub.stream[0].SetVolume(); VirtualDub.stream[0].SetCompression(); VirtualDub.stream[0].EnableFilterGraph(0); VirtualDub.stream[0].filters.Clear(); VirtualDub.video.DeleteComments(1); VirtualDub.video.AdjustChapters(1); VirtualDub.video.SetDepth(24,24); VirtualDub.video.SetMode(3); VirtualDub.video.SetFrameRate(0,1); VirtualDub.video.SetIVTC(0,0,-1,0); VirtualDub.video.SetRange(0,0); VirtualDub.video.SetCompression(0x64686663,0,10000,0); VirtualDub.video.SetCompData(20,"AAAAAAIAAAAAAAAAAQAAAAAAAAA="); VirtualDub.video.filters.Clear(); VirtualDub.subset.Clear(); VirtualDub.subset.AddRange(0, 547); VirtualDub.SaveAVI("C:\\Tools\\Working\\Short0001.avi"); VirtualDub.Close();