Compounded Question

[SuperTonic]

Hi, I'm new to TVP and I had a few observations I wanted to post... First here is my configuration:

Mac OS 10.6.2
2GHz Intel Core Duo
4 GB 667 MHz DDR2 SDRAM
TVP Animation 9.5.10

#1 When I check the app configuration/preference it displays the Physical Memory as 2047MB? This is a a text field that can't be edited. Does this mean that this is the Max amount of memory the App will take up, or does this mean that the program is really not seen acknowledging that I'm running 4 Gigs of RAM?

#2 When running the Video Grabber using a Firewire camera my processors spike in usage and stay pegged to +100% on both cores... is this a normal behavior?

#3 My instructors @ AAU recommended to purchase a separate USB Flash Drive to allow TVP to use it as a destination for it's /Temp folder. I'm wondering from a computer engineering perspective, how does this improve rendering and total software performance? I ask this, because this is the first time I've heard of any app taking and advantage of cache files stored outside of the root hard drive.

#4 And last but not least... as trivial as this might sound, I've noticed some sentimental crashes of the app; I call them sentimental, because as of right now I can't duplicate a particular behavior, other than clicking around and punching options like a maniac that doesn't know what's he is doing. On a more serious closure, what I meant to say is, when the Application Crashes unexpectedly and Apple generates a bug report log, are these logs later analyzed by the TVP staff?

Thanks in advance, and sorry for the load of subtopics within this post.

[Elodie]

#1 : It's only the memory cache, not the memory used by the software (the software sees all your Ram). Moreover, on a Mac OsX, the 4GB can be fully used by TVP Animation (contrary to Windows 32bits).

#2 : Nope, it's not normal.

#3 : Well, a Flash USB key is limited by the USB bus (max 20MB) and the Flash technology (5/10MB when writing), so I don't advise you to use this. You should better separate your files (don't make one full 10 minutes project in 5000x4000pixels with 10 layers). If you want to improve TVP Animation skill, well, you can change your CPU : TVP Animation only depends on the Ram, the cores and the free hard drive space. With your current Mac, you have 2 cores in 2Ghz, it means you have 4Ghz of CPU. It's sufficient, but if you want to create very heavy project, you should change your CPU for an octo core. And don't forget to free some space on your hard drive =)

#4 : That's strange, TVP Animation crashes really rarely... Can you identify how or when it crashes ? You can send us by email the report bug or what is written in your console. Crashes frequently happen when people allow to many cache memory to TVP Animation : there is no memory enough for the software.

[SuperTonic]

Thank you Elodie!

[Elodie]

you are welcome =)

[D.T. Nethery]

#3 : Well, a Flash USB key is limited by the USB bus (max 20MB) and the Flash technology (5/10MB when writing), so I don't advise you to use this. You should better separate your files (don't make one full 10 minutes project in 5000x4000pixels with 10 layers). If you want to improve TVP Animation skill, well, you can change your CPU : TVP Animation only depends on the Ram, the cores and the free hard drive space. With your current Mac, you have 2 cores in 2Ghz, it means you have 4Ghz of CPU. It's sufficient, but if you want to create very heavy project, you should change your CPU for an octo core. And don't forget to free some space on your hard drive =)

Hi, Elodie,

I have a question about creating the Temp Directory for TVP .

Somewhere along the line (maybe this is going back to the Mirage days in 2003) I have apparently picked up the erroneous information that a Temp Directory should be created using
an external drive. I can’t remember where I read that , but I have it in my notes .

But from your answer above it seems that I have been mistaken.


So then I am curious about what kind of external device is supposed to be used for a Temp Directory ? If USB is limited by the speed of the USB bus (max 20MB) that would also apply to USB external hard drive, correct ? So is it intended that the data storage device should be a Firewire device ? But this is not clear in the user manual. USB devices are common , Firewire drives much less common and many PC users do not have any firewire ports unless installed optionally. Apple appears to be phasing out firewire (new Macbook Pro only has one Firewire 800 port and regular Macbook does not have any firewire) . It is not easy to find a Firewire external drive . So most people will use a USB drive for external data storage.

The TVP User Manual section 1-14 says this about the Temp Directory:

“The faster the data storage device on which your temporary files are stored, the faster TVP Animation will work. Always ensure this storage device has sufficient available storage capacity.”

Therefore my question is : what exactly is the “data storage device” being referred to in the TVP User Manual if it is not an external drive ? Your answer above sounds like no external device needs to be used, but simply use the the hard drive on the user's computer.

Thanks.

-David Nethery

[Elodie]

But from your answer above it seems that I have been mistaken.

Well, maybe it existed in Mirage's time, but it's not the case anymore, now =)

So then I am curious about what kind of external device is supposed to be used for a Temp Directory ?

It can be a second internal hard drive, installed within the computer. In theory, a fast SSD device should be perfect.

[malcooning]

An SSD drive is nice, but it'll cost you the house, and it is still not the fastest solution.

You can achieve faster (much faster infact) access speed by using RAID.
RAID stands for Redundant Array of Inexpensive Drives. It means you can hook up many drives in a stripe (or in another method), which will all act as a single drive, with the capacity of the sum of capacities of all the striped drives combined, and with the speed of them combined. Thus, if you plug 2 hard drives of which 100 MB/sec, you get a near access speed of 200MB/Sec. The drives can be whatever cheap drives you find, which means that you can find very cheap drives.

If you have a desktop you can install both drives in your box easily. If you have laptop you might have an eSATA connection (which gives you a SATA connection externally. 10 times faster than USB), or you can buy an express card with eSATA ports, and then buy an external drives unit which can hold a couple of hard drives in RAID (there's plenty of these in many flavors). I personally prefer the former option and have 2 RAID sets in my desktop. One is a fast kind (RAID 0), on which I keep all media files as it gives me fast read/write, and another is mirrored (RAID1), which gives a slight speed improvement but mainly an automatic backup of everything I keep on it. I also have an external drive connected to my laptop via eSATA, and which gives me ability to read media off the drive without having to copy it to my internal drive.

The fastest solution available (as I very recently discovered thanks to Chad) is to allocate a portion of you system RAM and create a virtual drive of it. Say you have 8GB RAM, you can take 4GB and install a virtual drive onto it. this drive will be seen by your computer as a hard drive. You can then set your temp location to that drive. The speed of your RAM drive can be something of 100 times faster than a USB connection speed.

There are other solutions but mainly you should concern yourself with where the temp file goes, and where your media files go. Temporary files are used by the application and they tend to be as large as your current open projects are. This means that if you set a RAM drive like explained above, your main bottleneck in the system will be your CPU, not your storage. As for media files, it depends how you work. If you tend to preload video/sounds etc, this is written to the temporary file by TVP and you will suffice with a fast location for your temp files. But if you tend to run files without preloading (which is similar to what happens in compositing or editing application) you need a fast drive to read the media from. In this case a fast RAID (2,3,4 disks striped together) will give you the access speed to play the media back smoothly in high resolutions.

[D.T. Nethery]

So then I am curious about what kind of external device is supposed to be used for a Temp Directory ?

It can be a second internal hard drive, installed within the computer. In theory, a fast SSD device should be perfect.

Perhaps that could be clarified in the TVP User Manual , on how to set up a Temp Directory ?

I don't know about you , but when I read this phrase in the TVP user manual (1-14) :

"The faster the data storage device on which your temporary files are stored, the faster TVP Animation will work. Always ensure this storage device has sufficient available storage capacity.”

"data storage device" implies an external device , not a second internal drive.

What about for someone like me who uses a Macbook ? There is no room to install a second internal hard drive.

In theory, a fast SSD device should be perfect

Does TVP Development Co. recommended an SSD device to use for TVP's temp directory ? (brand name, model number?)

Does this mean that an external hard drive ( HDD) is NOT recommended as the temp directory ?

-------

*EDIT: thank you Malcooning for the detailed answer with alternatives to the SSD device. I am still trying to wrap my technically challenged little brain around what you are saying.

It seems odd to me that if this sort of thing is required for maximum efficiency when working in TVP it is not covered at all in the TVP User Manual. Just a few lines about designating a Temp Directory and :
"The faster the data storage device on which your temporary files are stored, the faster TVP Animation will work. Always ensure this storage device has sufficient available storage capacity.”


Not a clue about having to set up RAID, etc. How is anyone supposed to know this ?

[malcooning]

What about for someone like me who uses a Macbook ? There is no room to install a second internal hard drive.

What version of Macbook do you have?
do Macs have expansion slots (express card)?

Does this mean that an external hard drive ( HDD) is NOT recommended as the temp directory ?

Not via USB. It's better to store your files on the USB drive, and make room for the temp files on your internal drive. Even in it's slowest flavor, your internal drive is always faster than USB or Firewire.

[D.T. Nethery]

What about for someone like me who uses a Macbook ? There is no room to install a second internal hard drive.

What version of Macbook do you have?
do Macs have expansion slots (express card)?

Does this mean that an external hard drive ( HDD) is NOT recommended as the temp directory ?

Not via USB. It's better to store your files on the USB drive, and make room for the temp files on your internal drive. Even in it's slowest flavor, your internal drive is always faster than USB or Firewire.

I have a Macbook Pro 17" expansion: ExpressCard/34 slot.

(by the way: All other Macbook models do not have the Express Card expansion slot. )

That's interesting about how the internal drive should be used for the temp directory instead of a USB or Firewire external drive . I could have sworn that I had read somewhere that an external device was supposed to be used as the Temp Directory (and indeed that is what the wording in the User Manual in section 1-14 would lead one to believe --- " data storage device" doesn't sound like an internal drive, but rather an external drive.)
Am I crazy ? Did I dream this up out of nowhere or just totally misinterpret what the user manual is saying about "data storage devices" ?

[malcooning]

It seems odd to me that if this sort of thing is required for maximum efficiency when working in TVP it is not covered at all in the TVP User Manual. Not a clue about having to set up RAID, etc. How is anyone supposed to know this ?

This kind of knowledge is not something mentioned in any user manual of any media software. It's not up to the software provider to recommend such things, in case that something wrong happens and then the fault can be pinned at the software maker.

the knowledge about fast storage is readily available on the web, in much abundance, for some half a decade or so. Usually people are getting deeper into digital creation tend to search about, and read recommendations by other professionals, and sooner or later are bound to come across this bit of important knowledge. Just like you do now

I guess for people who use computer professionally, some things are considered common knowledge already. For example, if you pick up a USB Drive and it says USB 2.0, It's up to you to find out what is USB 2.0. and if you then find that it drives data 30MB/sec and it utilizes some percentage of your CPU while working, you can think of the size of your project and ask yourself whether your project can be played back smoothly at 30MB/sec, taking into consideration that there are other bits of information that will already take some of that 30MB/sec. So you can look at firewire 800, which will give more access speed, but not optimal. And so on.

think of it like you're buying a spade. It doesn't come with a booklet telling you where the soil is softer to dig.

Am I crazy ? Did I dream this up out of nowhere or just totally misinterpret what the user manual is saying about "data storage devices" ?

you're right. The wording is misleading (although not incorrect), and should be changed. I agree that also some recommendations for possible storage methods should be added.

I have a Macbook Pro 17" expansion: ExpressCard/34 slot.?

I then seriously recommend you to search for an ExpressCard with eSATA ports, and a drive with eSATA interface (there should be many of them. just don't buy from the Mac store, they are USBcentric and they overprice). A good example of drives with eSATA are the MyBook series by western Digital, although not the cheapest. With eSATA you'll get the same access speed as you would with your internal drive.

[D.T. Nethery]

It seems odd to me that if this sort of thing is required for maximum efficiency when working in TVP it is not covered at all in the TVP User Manual. Not a clue about having to set up RAID, etc. How is anyone supposed to know this ?

This kind of knowledge is not something mentioned in any user manual of any media software. It's not up to the software provider to recommend such things, in case that something wrong happens and then the fault can be pinned at the software maker.

Yes, that makes sense (to avoid any blame for user errors) , BUT ... since the Temp Directory is an integral part of the TVP set up and is necessary (?) for running TVP for maximum efficiency I think there needs to be some mention in the user manual of the options available for "data storage devices" , which to many of us "data storage device" = "USB drives" , because USB drives are the most commonly available. (although from what you're telling me it doesn't matter if the external device is USB or the faster Firewire , because we shouldn't be using external devices for Temp Directory anyway , even though TVP user manual says the temp drive should be set to a "data storage device" , not mentioning using an internal drive. You see why I'm confused ?)

Thanks again for all your help in clarifying this for me.

[slowtiger]

Just for the record: I successfully use external Firewire drives on my Mac with TVP.

[D.T. Nethery]

Just for the record: I successfully use external Firewire drives on my Mac with TVP.

Hi, Slowtiger,

Yeah, my temp directory is an external Firewire drive and I thought it was working fine all this time, but now I don't know ? I've had no complaints about how my TVP runs using the external firewire device .

The reason for my extended discussion here regarding "data storage devices" and what that phrase means to the average person is because the students at my school generally balk at using external Firewire drives because they are difficult to find and more expensive , so for some reason I thought it would be ok to use external USB drive as the Temp Directory because USB is what most have access to , but it turns out I was mistaken to recommend USB devices because they are too slow. Instead I am finding out that all sensible people know that a second internal drive or a RAID or SSD device should be used , but this is not hinted at in the TVP User Manual , only the mention of a "data storage device" to be used for the Temp Directory, which I interpreted as meaning an external storage device, but it should actually be an internal drive. I think I'm finally getting it ... sort of.

As everyone can now tell from my sputtering questions (if you didn't know before ) I know next to nothing about how computers actually function on the inside. That's why I use a Mac I suppose: I turn it on , it works. Simple. I can spend most of my time drawing and animating and I don't have to think about it. I don't really care how it works. Now I am paying for my folly.

(but yes, my external Firewire drive has seemed to function just fine as the temp directory , so I am still a little confused ... perhaps if I switch to using an ExpressCard with eSATA ports and a drive with eSATA interface I will suddenly see the light and realize what I've been missing all these years in terms of speed ? )

[malcooning]

perhaps if I switch to using an ExpressCard with eSATA ports and a drive with eSATA interface I will suddenly see the light and realize what I've been missing all these years in terms of speed ? )

Basically, if it works for you, and you have no reason to complain, don't change anything. Sometimes knowing more and attempting more just opens more windows for wasps to nest in.

But if you want to test the difference, let's begin with this:
1) Describe a typical project of yours. What is the project dimension, how many layers, what are the file sizes, and how many frames/length in time.
2) When you playback your animation, does it hitch here in there? do you get all frames playing real time? when you scrub, does the sound play smoothly and playhead is responsive?