There are several things you need to consider when quitting any psychotropic drug, tropicdonna.
Firstly, discuss this with the prescribing doctor, if for no other reason than you may need to have diazepam in smaller dose tablets. If s/he recommends a different approach to the following then follow your doctor's advice.
Secondly, how are you going to deal with whatever problem you were taking the drug(s) for? Have you and your physician developed a strategy - another drug or psychotherapy to cope with a possible return of the anxiety? If not then I urge you to do this before proceeding. Often what are perceived as symptoms of withdrawal are really just a return of the anxiety.
As for the specifics of quitting diazepam, benzodiazepine (and many other medications including antidepressants, beta blockers, antibiotics after long-term use, even aspirin if taken daily for more than a few weeks) should never be quit 'cold-turkey'! The brain has adapted to its presence including down-regulating the number of benzo/GABA complex receptors and if you quit suddenly this will often provoke a reaction until an adaption to the new circumstance occurs.
Fortunately, Valium is probably the easiest of the benzodiazepines to withdraw from because of its long half-life - about 50-200 hours - hence your lack of symptoms until 7 days after quitting.
I suggest you reduce the dose by no more than 1mg every week or two until you're down to 1 mg then cut back to 0.5mg/day. Buying a pill-cutter will make this easier/more accurate (and less bloody, IME) than using a knife. I assume you are being prescribed 5mg tablets so suggest you ask your physician for 2mg tablets. Diazepam is also available in liquid form which is ideal if you need to reduce the dose in smaller increments.
Once you've down to 0.5mg for a week try quitting it completely. If that doesn't work, instead of reducing the dose further on a daily basis, it is often better to only take 0.5mg every second day for a week then every third day, fourth, and finally fifth day after which you should be able to stop.
The above may sound like a long process. While some are particularly sensitive, most people can do it much quicker than I've suggested without drama. But if you've been taking diazepam for 40 years (actually you been taking it all your life virtually from conception as it naturally occurs in all foods) what's the hurry, huh?
Finally, there is probably more written about benzodiazepine withdrawal than for any other group of drugs, including illicit ones and much of it is very dubious. Many of the effects blamed on withdrawal are often just a return of the underlying condition. Withdrawal is rarely an issue in the other patient cohort to have used benzodiazepines: epileptics. The vast majority of these were switched to non benzodiazepine anti seizure medications when these became available in the 1980/90s with few problems even though many had been taking them in huge doses, often the equivalent of several hundred milligrams of diazepam daily. Interestingly, a number of studies have found that anxiety patients will often experience withdrawal effects when told their benzodiazepine dose has been reduced even though no reduction occurred.
NOTE: I am not a doctor, and more importantly, I'm not your doctor so the above advice is necessarily of a general nature which may or may not apply to you. Always consult your health provider before acting on anything you've read here.